Bleeding Edge CF Architecture

We just deployed a new bleeding edge CF architecture last month. I’m going to be blogging about it, and a lot more this year. Stay tuned!

Hiring Woes in the ColdFusion World

No I’m not going to speak about the lack of CF developers. Quite the opposite, I think there are plenty out there. The quality? Well, that’s what we’ll mention here.

Over the past 8 years, at my current position as an executive who also develops actively, I’ve reviewed probably over 300+ resumes, and phone interviewed over 200+ CF folks alone from all over the world. There are some things you learn along the way as a hiring manager – how the market for talent works, the general capabilities of CF developers, and what their typical strengths and weaknesses look like.

I did want to mention two recent resumes and hiring experiences that both went south. This is advice for all CF folks, as I’ve waited to write this to make sure nothing I said would be personal or negative, rather that I could put a positive spin on things and give advice.

As a developer, there really needs to be a focus on self-improvement. In most cases, I don’t find that – I’m surprise how many people “settle.” For me ColdFusion enables each and every day to accomplish an overall vision I have for the projects that I work on – ColdFusion is not the goal in and of itself, nor is the pay check that drives me. I couldn’t work in an environment which gave me access to ColdFusion, but restricted my ideas on how to improve the projects I worked on, or paid me a high salary – but denied me a voice.

Very few developers (roughly 2% from experience) have anything to show for self-improvement. Have they kept up on blogs, read books, attended a conference, tried new techniques, improved their understanding of object-oriented (if not aspect-oriented) programming? What drives them? The answer I usually get is that they are interested in new techniques and approaches, it’s just that their work environment never provided them the opportunity to do so. This to me is a red flag – I expect any serious developers to have their own work environment at home, and work on their own pet projects on the side.

But I tend to forgive the developers, and blame the management in those companies for not fostering innovation. How often do folks meet to exchange ideas in your department, present on new findings, and are given time to develop new ideas on their own? As a manager, I think less about micro-managing or macro-managing, but finding opportunities for my folks to shine. I want to always market my team to the rest of the company as a high performing, well motivated team.

And that’s what sets our company apart, and that is also what leads to problems in hiring. I often talk, in the first round of interviews, about my philosophy, how the team is run, and how we’re simply different. This gets people VERY excited and I get them thinking about how they can contribute to this environment, what skills they bring, and what they’d like to learn. I often tell them that they can expect the first 90 days to be like a tidal wave – we use so many bleeding edge ColdFusion techniques (the rights ones, not all), that it can be very overwhelming.

The problem occurs when people come with years of experience, but that doesn’t translate to necessarily years of accomplishments. You may have worked for 12 years as a ColdFusion developers, in senior roles even, but if you have never touched a framework, then that is a problem. One year of experience here, I often say, is worth 3-5 at other places.

If you are serious at getting better as a developer, then you have to find an environment that is going to support (and push) you. That in itself is worth a lot.

And that’s where the problem is. I recently got two resumes from folks – who each had 12+ years of CF development experience, but I could not say they were truly senior (in my perspective). Regardless, I was willing to offer opportunities to them as Senior folks, with the understanding that they would be able to pick up all the exciting things I talked about.

And then… the focus turned to salary. Do you want to guess what both of them were asking for? Well, here it is – $130K-$135K annually plus benefits.

I was dumbfounded. I am no position to offer that kind of salary to anyone. Heck, even I don’t make that kind of money, and I’m a Director! You have to do your research: we’re a small company (less than 50 folks), and you have to add the value given by a supportive work environment.

I’m not saying we can’t attempt to meet your needs, but seriously… You know that old saying – you dress for the position you want, not the one you have? Well, that applies to CF developers as well. If you feel you’re worth a ton of money, then exhibit the qualities of a world-class developer. If I asked you to rate yourself on the scale of journeyman to master – where would you fall? Do you have experience leading a sophisticated team – often times filled with people who are smarter than you? No. Have you worked with OO frameworks in CF for 9 years like me? No. (I celebrate my 9 year anniversary with Mach II next month!) Are you a master of both the back-end and front-end languages? No. Do you have the communication skills to interface with all sorts of different stakeholders? No.

So what then qualifies you to ask for that much? Well – the answer usually is: “I can make that much consulting.” Well, if you want to pay for your own health insurance, forgo benefits, and work in an environment that really is not going to make you a better developer, and for a company that isn’t truly committed to ColdFusion as a development platform of choice, then go ahead. But if you want to earn a real salary, and be given the opportunity to prove yourself, and work your way up the ladder, then I’m here to support you and am all for that. But be real.

There is nothing at my company that say that you can’t make that kind of money, with benefits and possible bonuses, it’s quite possible. But you’ve got to also have the pedigree and focus to enable that. If you’re overly focused on salary, then I’m sorry – I have something to offer that is worth so much more.

There is probably more to say, but I’ll stop here. I’ve probably already said too much.

Seven Year Itch – Over and Done!

Back in California this week. Look forward to blogging again!

ColdFusion 10 Released

In case you didn’t know, ColdFusion 10 was released today. More information can be found @ http://www.adobe.com/products/coldfusion-family.html.

Big Change in the jQuery World

jQuery made a big announcement today! Looks like good news. Now if they could only get the plugins site back up! Jeez, its been long enough already!

Read more @ http://blog.jquery.com/2012/03/06/announcing-the-jquery-foundation/.

Breaking Radio Silence

These past 4 months have been quite arduous.
We launched a whole new upgrade to our ColdFusion-based SaaS product that included a new UX/UI re-do (10 month project!), moving from Verity to Solr (6 months!), and so much more.
After many restless night, numerous 60-hour weeks, I’m back. Will be posting more shortly!

Git Deployer in ColdFusion – Need Help!

We have settled really well on Git, and have branches for all developers and each environment, like I described earlier.
However, what we don’t have is an easy way to deploy a git branch, or simply execute a git pull command remotely. I don’t want to do it using SSH and Git BASH, which seems to be what Google turns up.
I looked to see if I could whip up a home grown ColdFusion based solution. I looked at Git.cfc, however it lack any teeth. You can execute a “git pull” command, but the code doesn’t have any mechanism to load git keys, so the code errors out. I also looked at the jGit JAR file, and interfacing with that, but there is no documentation – only a link to forums.
Has anyone built anything that you found useful, or if you use Git, how do you effectively issue an update command for example, to a dev or QA server?

Like JSLint? Try JSHint!

JSHint is a new fork of JSLint.
To quote them:
“JSHint is a fork of Douglas Crockford’s JSLint that is designed to be more flexible than the original. Our goal is to make a tool that helps you to find errors in your JavaScript code and to enforce your favorite coding style. We realize that people use different styles and conventions, and we want our tool to adjust to them. JSHint will never enforce one particular convention. JSHint is developed and supported by the JavaScript developer community and we welcome feedback from everybody who cares about the language.”
We’re definitely going to start working on auditing and enforcing JS code through a mechanism like JSHint.

Lessons Learned: Moving from Verity to Solr (Part 6)

Previously, in Part 5 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
If you note Part 4 in this series, you’ll see that I speak about upgrading to Solr 1.4.1. Well, for me, that wasn’t enough. Allow me to explain in a story I like to call: 2 JARs and a WAR.
You’ll note that CF supports two request handlers – standard and dismax. In the CFSearch tag, you specify “type”. Typically, all you ever need is ‘standard’, which supports wildcards all the good stuff Solr comes with out of the box. But there is ONE BIG caveat, boosting is busted.
What is boosting? Lets say you are indexing three fields, author, summary, keywords. You definitely want matches against keywords to give a higher score- basically its weight to be greater than author for example. According to the Solr book (the only one on 1.4!), you need to specify what the weights are during initial indexing – in a feature called index time boosts. Solr seems to be sly about not telling you this outright:
“At index-time, you have the option to boost a particular document (entirely or just a field). This is internally stored as part of the norms number, which must be enabled for this to work. It’s uncommon to perform index-time boosting.
At query-time, we have described earlier how to boost a particular clause of a query higher or lower if needed. Later the powerful Disjunction-Max (dismax for short) query will be demonstrated, which can apply searches to multiple fields with different boosting levels automatically.”
Basically, for some odd reason, you cannot boost upon search. In fact, one of the features of dismax is “Searches across multiple fields with different boosts through Lucene’s DisjunctionMaxQuery.” What’s odd is that dismax is somewhat deprecated and everything seems to reference ‘standard.’
You would think at this point, no big deal right, I’ll just switch from standard to dismax and thats it. Wrong. What the docs also don’t clearly spell out is that with dismax, wildcards are NOT supported. That’s correct. NOT!
So you have some features in standard, which are critical for free-text searching, but also you need proper scoring and boosting, which is available in dismax. To make matters worse, if you search with “polymer*” as an example, the wildcard at the end in dismax throws a CF error:
“String_index_out_of_range_1__javalangStringIndexOutOfBoundsException_String_index_out_of_range_1___at_javalangAbstractStringBuilderreplaceAbstractStringBuilderjava797___at_javalangStringBuilderreplaceStringBuilderjava271___at_orgapachesolrhandlercomponentSpellCheckComponenttoNamedListSpellCheckComponentjava458___at_orgapachesolrhandlercomponentSpellCheckComponentprocessSpellCheckComponentjava142___at_orgapachesolrhandlercomponentSearchHandlerhandleRequestBodySearchHandlerjava195___at_orgapachesolrhandlerRequestHandlerBasehandleRequestRequestHandlerBasejava131___at_orgapachesolrcoreSolrCoreexecuteSolrCorejava1321___at_orgapachesolrservletSolrDispatchFilterexecuteSolrDispatchFilterjava341___at_orgapachesolrservletSolrDispatchFilterdoFilterSolrDispatchFilterjava244___at_orgmortbayjettyservletServletHandler$CachedChaindoFilterServletHandlerjava1089___at_orgmortbayjettyservletServletHandlerhandleServletHandlerjava365___at_orgmortbayjettysecuritySecurityHandlerhandleSecurityHandlerjava216___”
And if you use the Solr Admin interface, it doesn’t throw an error, so its almost impossible to figure out that the wildcard doesn’t work in dismax, because Solr by itself doesn’t throw an error!
Needless to say, at this point, things looked pretty hopeless. I felt like Solr wasn’t going to make it. I had a simple use case, and Solr couldn’t handle it without some big compromises.
So I decided to take a leap. Heck, I had already upgraded from Solr 1.4 to 1.4.1, why not something more recent? I knew from my digging around that Solr 1.5 in currently in development, and so is Solr 3.0, and Solr 4.0! More on that later.
I decided to upgrade to Solr 1.5 Dev. Why? Well, in reading the change log, something caught my eye. There was a new request handler submitted by Lucid (more on that later as well!). It was called “edismax” and it had everything standard has, and everything dismax has put together along with a ton of improvement. Could this be? Could this work!?
So I decided to give it a try. Here are some steps.

1. Stop Solr Service
2. Build Solr 1.5 Dev from Maven (thanks to Joseph Lamoree for help there)
3. Like in Step 4, copy the WAR over
4. Stop CF Instance / Service.
5. Since this is a major version upgrade, you also needed to copy over the Solr Core and SolrJ JARs to <instance>cfusion-earcfusion-warWEB-INFcfusionlib replacing apache-solr-core.jar and apache-solr-solrj.jar respectively.

Now start up CF and Solr. You are running the Solr 1.5 Dev WAR, and CF is using the newer JAR files to communicate to Solr 1.5.
Now, in your SolrConfig.xml file (see earlier notes), added a new requesthandler. I am pasting a bare bones version below.
  <requestHandler name=”edismax” class=”solr.SearchHandler” >
    <lst name=”defaults”>
        <str name=”defType”>edismax</str>
        <str name=”echoParams”>explicit</str>
        <float name=”tie”>0.01</float>
        <str name=”qf”>custom1^0.1 custom2^8.0 custom3^2.0 custom4^10.0 key^1.5 uid^1.5 title^8.0 contents^2.0</str>
        <str name=”pf”>custom1^0.1 custom2^8.0 custom3^2.0 custom4^10.0 key^1.5 uid^1.5 title^8.0 contents^2.0</str>
        <str name=”fl”>*,score</str>
        <str name=”mm”>2&lt;-1 5&lt;-2 6&lt;90%</str>
        <int name=”ps”>100</int>
        <str name=”q.alt”>*:*</str>
        <!– example highlighter config, enable per-query with hl=true 
        <str name=”hl.fl”>summary title</str> –>  
        <str name=”spellcheck”>false</str>
        <!– omp = Only More Popular –>
        <str name=”spellcheck.onlyMorePopular”>false</str>
        <!– exr = Extended Results –>
        <str name=”spellcheck.extendedResults”>false</str>
        <!–  The number of suggestions to return –>
        <str name=”spellcheck.count”>5</str>
    </lst>
    <arr name=”last-components”>
        <str>spellcheck</str>
    </arr>
  </requestHandler>
You’ll note a couple things. First, defType is set to edismax and it is also using a new class. qf and pf (you can read in the Solr docs what they mean) is where I am specifying which fields need to be boosted at search time, and VERY importantly, all spellcheck options have to be “false”. I learned this the hard way. You can still do spell checking and return back the suggestions from CFSearch, but they must be false for wildcards to work, at least as the defaults in SolrConfig.xml.
Next, inside CFSearch, set type to edismax, and this may be a side note, but I was still getting errors, so I added URLDecode(criteria) where I was passing criteria to CFSearch, and voila!
Now I was running Solr 1.5 Dev 64-bit, CF was communicating with no errors, I was getting all the right scores, I could do trailing and prefixed wildcards, and I also no longer needed to do NGraming, so my collection size shrank. Woo-hoo!
To be continued…

Lessons Learned: Moving from Verity to Solr (Part 4)

Previously, in Part 3 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
This is a big one. Upgrading to the latest Solr. The Solr that comes with Adobe ColdFusion 9.0.1 is a slightly customized (from what I can tell) version of a Solr pre-1.4.0 release. It’s almost a year old!
I was having some trouble with some of my custom enhancements, so i decided to upgrade to Solr 1.4.1.
Before I dive into details, I have to give a shout to Vinu Kumar and Kunal Saini (Adobe Engineers) who confirmed some details and pointed me in the right direction.

1. Shut down the Solr Search Service (obvious)
2. You’ll notice in your $coldfusionSolrInstalldirwebapps dir, there is a WAR file called solr.war. Back this file up somewhere outside this directory.
3. Download latest Solr zip. Go to apache-solr-1.4.1dist. Notice that apache-solr-1.4.1.war file? Copy to the above directory and rename to solr.war.
4. Delete all files in the $coldfusionSolrInstalldirwork directory. I believe this is where it expands the WAR’s files.
5. Start the Solr Search service.

That’s it! You should see new files in the $coldfusionSolrInstalldirwork directory now. But there has to be a catch, right?
Yes there is, but its a small one. Creating new Solr collections via CFAdmin will fail. Why? Because its looking for those tweaks. Is there a workaround? Yes!
And I believe the following work around is reasonable, as most people will not be creating collections all the time. Usually they are created, and its set.
So, all you must do is the following:

1. Think of a new collection name. Easy.
2. When you create a Solr collection (or a Verity one), CFAdmin asks for a “path”. Note that path.
3. Copy $coldfusionSolrInstalldirmulticore emplateconf*.* to <path><new collection name>conf*.*
4. Now go to CFAdmin, point it to path=<path> with name=<new collection name>, and voila!

Essentially what happens here, is that once it notices a conf directory exists (which has the CF customization) it no longer freaks out.
That’s it. I’m running Solr 1.4.1 on Windows 2008 R2 64-bit! This is so awesome since I didn’t need to make any other changes, such as JVM settings, and other tweaks, they all carried over!