Like JSLint? Try JSHint!

JSHint is a new fork of JSLint.
To quote them:
“JSHint is a fork of Douglas Crockford’s JSLint that is designed to be more flexible than the original. Our goal is to make a tool that helps you to find errors in your JavaScript code and to enforce your favorite coding style. We realize that people use different styles and conventions, and we want our tool to adjust to them. JSHint will never enforce one particular convention. JSHint is developed and supported by the JavaScript developer community and we welcome feedback from everybody who cares about the language.”
We’re definitely going to start working on auditing and enforcing JS code through a mechanism like JSHint.

Git Deployer in ColdFusion – Need Help!

We have settled really well on Git, and have branches for all developers and each environment, like I described earlier.
However, what we don’t have is an easy way to deploy a git branch, or simply execute a git pull command remotely. I don’t want to do it using SSH and Git BASH, which seems to be what Google turns up.
I looked to see if I could whip up a home grown ColdFusion based solution. I looked at Git.cfc, however it lack any teeth. You can execute a “git pull” command, but the code doesn’t have any mechanism to load git keys, so the code errors out. I also looked at the jGit JAR file, and interfacing with that, but there is no documentation – only a link to forums.
Has anyone built anything that you found useful, or if you use Git, how do you effectively issue an update command for example, to a dev or QA server?

Lessons Learned: Moving from Verity to Solr (Part 7)

In our Verity days, we used a UDF called VerityClean (still available at CFLib), that did a lot of grunt work of cleaning keywords for Verity. In fact, if you read the description of the UDF, it says: “strips all invalid characters and word combinations from a search strign
to prevent verity from crashing.” Awesome, right?
Well, in moving to Solr, there was no equivalent. Solr can be very picky, it rocks when you have a UDF that:

• Replaces comma with OR
• Strips double spaces
• Strips bad characters
• Cleans up sequences of space characters
• Uppercases Solr terms like AND, OR, etc.

I just submitted SolrClean and a sister UDF uCaseWordsForSolr to CFLib. Enjoy!

UPDATE: The submissions to CFLib were never approved or simply disappeared. I’ll bring it to GitHub.

UPDATE 2: The submissions are now available on CFLib. I am preparing a separate post on them.

Breaking Radio Silence

These past 4 months have been quite arduous.
We launched a whole new upgrade to our ColdFusion-based SaaS product that included a new UX/UI re-do (10 month project!), moving from Verity to Solr (6 months!), and so much more.
After many restless night, numerous 60-hour weeks, I’m back. Will be posting more shortly!

We’re Hiring in SoCal Again!

We’re looking for good senior folks. More details @ http://www.ecivis.com/our-company/careers/.

Let me know if you have any questions! We’re open for mid-, sr, architect, or even manager position!

Sr. Web Application Developer – ColdFusion

Location: Pasadena, CA
Telecommute: Not available
Salary: DOE
Relocation: Available

eCivis, the market leader in providing web-based solutions to local
governments (software as a service) is seeking an advanced-level
developer who lives to push ColdFusion and object-oriented web
development to the edge. If you have experience in working with
frameworks and combining new technologies with solid development
practices, we want to talk to you. Join our growing team as part of
the product management and software engineering departments building
innovative web-based enterprise applications. This position directly
interacts with all departments and significantly impacts our client
base. Learn more about us at www.ecivis.com.

JOB DESCRIPTION:
The Sr. Web Developer / Architect will drive the direction of the
company’s technology solutions, define and implement best practices
and build lean, powerful and extensible functionality. This position
offers significant responsibility and growth potential. You will work
closely with key stakeholders and wear many hats.

RESPONSIBILITIES:
•       Assist in the design and development of object-oriented web applications based upon best practices.
•       Ensure proper operation and intelligent enhancements of existing web-based products and services.
•       Ensure development consistency and strive for continued development innovation.
•       Maintain best practices to meet department goals and assist in adapting the department to future changes in technology.
•       Help assess risk related to technology and resources, and assists in minimizing that risk.

IDEAL CANDIDATE SHOULD POSSESS THE FOLLOWING SKILLS:
•       Passionate about Creating Great Products and Innovative Features
•       Driven and Realistic: Good estimator of time and effort
•       Proactive about Self-Improvement and acquiring new knowledge and skills
•       Motivated team player and ability to lead independent self-managed projects
•       5+ years developing complex ColdFusion applications
•       Advanced knowledge of ColdBox, Mach II, Model Glue or other OO Frameworks
•       Understanding of ColdFusion Best Practices and Design Patterns
•       Solid communication skills: Written and Verbal
•       Demonstrated documentation and code-testing skills
•       Strong understanding of complex SQL database-driven products
•       Strong knowledge of JavaScript libraries and CSS
•       Obsessive problem-solving and debugging skills

PREFERRED QUALIFICATIONS:
•       Product Development Experience
•       Git and/or Subversion Experience
•       Active in open source projects
•       Understands scaling and performance

If you meet these qualifications, we invite you to submit a resume and
cover letter, including a complete salary history via e-mail:
technology <at> ecivis.com.
Please no phone calls or postal mail. Only qualified candidates will be contacted.

eCivis, Inc., is an equal opportunity employer. This position is for full-time employment in Pasadena, California.

Lessons Learned: Moving from Verity to Solr (Part 6)

Previously, in Part 5 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
If you note Part 4 in this series, you’ll see that I speak about upgrading to Solr 1.4.1. Well, for me, that wasn’t enough. Allow me to explain in a story I like to call: 2 JARs and a WAR.
You’ll note that CF supports two request handlers – standard and dismax. In the CFSearch tag, you specify “type”. Typically, all you ever need is ‘standard’, which supports wildcards all the good stuff Solr comes with out of the box. But there is ONE BIG caveat, boosting is busted.
What is boosting? Lets say you are indexing three fields, author, summary, keywords. You definitely want matches against keywords to give a higher score- basically its weight to be greater than author for example. According to the Solr book (the only one on 1.4!), you need to specify what the weights are during initial indexing – in a feature called index time boosts. Solr seems to be sly about not telling you this outright:
“At index-time, you have the option to boost a particular document (entirely or just a field). This is internally stored as part of the norms number, which must be enabled for this to work. It’s uncommon to perform index-time boosting.
At query-time, we have described earlier how to boost a particular clause of a query higher or lower if needed. Later the powerful Disjunction-Max (dismax for short) query will be demonstrated, which can apply searches to multiple fields with different boosting levels automatically.”
Basically, for some odd reason, you cannot boost upon search. In fact, one of the features of dismax is “Searches across multiple fields with different boosts through Lucene’s DisjunctionMaxQuery.” What’s odd is that dismax is somewhat deprecated and everything seems to reference ‘standard.’
You would think at this point, no big deal right, I’ll just switch from standard to dismax and thats it. Wrong. What the docs also don’t clearly spell out is that with dismax, wildcards are NOT supported. That’s correct. NOT!
So you have some features in standard, which are critical for free-text searching, but also you need proper scoring and boosting, which is available in dismax. To make matters worse, if you search with “polymer*” as an example, the wildcard at the end in dismax throws a CF error:
“String_index_out_of_range_1__javalangStringIndexOutOfBoundsException_String_index_out_of_range_1___at_javalangAbstractStringBuilderreplaceAbstractStringBuilderjava797___at_javalangStringBuilderreplaceStringBuilderjava271___at_orgapachesolrhandlercomponentSpellCheckComponenttoNamedListSpellCheckComponentjava458___at_orgapachesolrhandlercomponentSpellCheckComponentprocessSpellCheckComponentjava142___at_orgapachesolrhandlercomponentSearchHandlerhandleRequestBodySearchHandlerjava195___at_orgapachesolrhandlerRequestHandlerBasehandleRequestRequestHandlerBasejava131___at_orgapachesolrcoreSolrCoreexecuteSolrCorejava1321___at_orgapachesolrservletSolrDispatchFilterexecuteSolrDispatchFilterjava341___at_orgapachesolrservletSolrDispatchFilterdoFilterSolrDispatchFilterjava244___at_orgmortbayjettyservletServletHandler$CachedChaindoFilterServletHandlerjava1089___at_orgmortbayjettyservletServletHandlerhandleServletHandlerjava365___at_orgmortbayjettysecuritySecurityHandlerhandleSecurityHandlerjava216___”
And if you use the Solr Admin interface, it doesn’t throw an error, so its almost impossible to figure out that the wildcard doesn’t work in dismax, because Solr by itself doesn’t throw an error!
Needless to say, at this point, things looked pretty hopeless. I felt like Solr wasn’t going to make it. I had a simple use case, and Solr couldn’t handle it without some big compromises.
So I decided to take a leap. Heck, I had already upgraded from Solr 1.4 to 1.4.1, why not something more recent? I knew from my digging around that Solr 1.5 in currently in development, and so is Solr 3.0, and Solr 4.0! More on that later.
I decided to upgrade to Solr 1.5 Dev. Why? Well, in reading the change log, something caught my eye. There was a new request handler submitted by Lucid (more on that later as well!). It was called “edismax” and it had everything standard has, and everything dismax has put together along with a ton of improvement. Could this be? Could this work!?
So I decided to give it a try. Here are some steps.

1. Stop Solr Service
2. Build Solr 1.5 Dev from Maven (thanks to Joseph Lamoree for help there)
3. Like in Step 4, copy the WAR over
4. Stop CF Instance / Service.
5. Since this is a major version upgrade, you also needed to copy over the Solr Core and SolrJ JARs to <instance>cfusion-earcfusion-warWEB-INFcfusionlib replacing apache-solr-core.jar and apache-solr-solrj.jar respectively.

Now start up CF and Solr. You are running the Solr 1.5 Dev WAR, and CF is using the newer JAR files to communicate to Solr 1.5.
Now, in your SolrConfig.xml file (see earlier notes), added a new requesthandler. I am pasting a bare bones version below.
  <requestHandler name=”edismax” class=”solr.SearchHandler” >
    <lst name=”defaults”>
        <str name=”defType”>edismax</str>
        <str name=”echoParams”>explicit</str>
        <float name=”tie”>0.01</float>
        <str name=”qf”>custom1^0.1 custom2^8.0 custom3^2.0 custom4^10.0 key^1.5 uid^1.5 title^8.0 contents^2.0</str>
        <str name=”pf”>custom1^0.1 custom2^8.0 custom3^2.0 custom4^10.0 key^1.5 uid^1.5 title^8.0 contents^2.0</str>
        <str name=”fl”>*,score</str>
        <str name=”mm”>2&lt;-1 5&lt;-2 6&lt;90%</str>
        <int name=”ps”>100</int>
        <str name=”q.alt”>*:*</str>
        <!– example highlighter config, enable per-query with hl=true 
        <str name=”hl.fl”>summary title</str> –>  
        <str name=”spellcheck”>false</str>
        <!– omp = Only More Popular –>
        <str name=”spellcheck.onlyMorePopular”>false</str>
        <!– exr = Extended Results –>
        <str name=”spellcheck.extendedResults”>false</str>
        <!–  The number of suggestions to return –>
        <str name=”spellcheck.count”>5</str>
    </lst>
    <arr name=”last-components”>
        <str>spellcheck</str>
    </arr>
  </requestHandler>
You’ll note a couple things. First, defType is set to edismax and it is also using a new class. qf and pf (you can read in the Solr docs what they mean) is where I am specifying which fields need to be boosted at search time, and VERY importantly, all spellcheck options have to be “false”. I learned this the hard way. You can still do spell checking and return back the suggestions from CFSearch, but they must be false for wildcards to work, at least as the defaults in SolrConfig.xml.
Next, inside CFSearch, set type to edismax, and this may be a side note, but I was still getting errors, so I added URLDecode(criteria) where I was passing criteria to CFSearch, and voila!
Now I was running Solr 1.5 Dev 64-bit, CF was communicating with no errors, I was getting all the right scores, I could do trailing and prefixed wildcards, and I also no longer needed to do NGraming, so my collection size shrank. Woo-hoo!
To be continued…

Lessons Learned: Moving from Verity to Solr (Part 5)

Previously, in Part 4 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
In Part 3 of this series, I spoke about my frustration with wild cards. Solr, out of the box, will support wild cards like ? or * anywhere in the criteria, except as the first character. In Part 3, I detailed a known work-around.
Well, to my amazement, in pouring over Solr docs as I have been doing, I found with Solr 1.4 there is a way to do this natively! Woo-hoo!
In my schema.xml, for your fieldType (in my case fieldType name=”text”…, under analyzer type=”index”), you can add a new filter called: ReversedWildcardFilterFactory. More details @ http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ReversedWildcardFilterFactory.
So I added: <filter class=”solr.ReversedWildcardFilterFactory” withOriginal=”true” maxPosAsterisk=”3″ maxPosQuestion=”2″ maxFractionAsterisk=”0.33″/>
Basically what this does is it takes the fields of type Text when indexing, and stores a reverse of them in the index. You can find what the additional parameters mean online, but basically setting withOriginal = true says to keep the original when indexing.
So let’s give an example. Let’s say you are indexing the word “bookshelf”. Before this setting, you could search for “book*” and this record would show just fine. But you could not search for “*shelf”. Now when you index, Solr will store “bookshelf” and “flehskoob”. When you search for “*shelf”, it takes note of the leading *, reverses it to “flehs*”, which would match “flehskoob”, and bingo, you have a match!
Note: Since you are storing additional data, obviously this will cause your collection size to grow, but its worth it!

Lessons Learned: Moving from Verity to Solr (Part 4)

Previously, in Part 3 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
This is a big one. Upgrading to the latest Solr. The Solr that comes with Adobe ColdFusion 9.0.1 is a slightly customized (from what I can tell) version of a Solr pre-1.4.0 release. It’s almost a year old!
I was having some trouble with some of my custom enhancements, so i decided to upgrade to Solr 1.4.1.
Before I dive into details, I have to give a shout to Vinu Kumar and Kunal Saini (Adobe Engineers) who confirmed some details and pointed me in the right direction.

1. Shut down the Solr Search Service (obvious)
2. You’ll notice in your $coldfusionSolrInstalldirwebapps dir, there is a WAR file called solr.war. Back this file up somewhere outside this directory.
3. Download latest Solr zip. Go to apache-solr-1.4.1dist. Notice that apache-solr-1.4.1.war file? Copy to the above directory and rename to solr.war.
4. Delete all files in the $coldfusionSolrInstalldirwork directory. I believe this is where it expands the WAR’s files.
5. Start the Solr Search service.

That’s it! You should see new files in the $coldfusionSolrInstalldirwork directory now. But there has to be a catch, right?
Yes there is, but its a small one. Creating new Solr collections via CFAdmin will fail. Why? Because its looking for those tweaks. Is there a workaround? Yes!
And I believe the following work around is reasonable, as most people will not be creating collections all the time. Usually they are created, and its set.
So, all you must do is the following:

1. Think of a new collection name. Easy.
2. When you create a Solr collection (or a Verity one), CFAdmin asks for a “path”. Note that path.
3. Copy $coldfusionSolrInstalldirmulticore emplateconf*.* to <path><new collection name>conf*.*
4. Now go to CFAdmin, point it to path=<path> with name=<new collection name>, and voila!

Essentially what happens here, is that once it notices a conf directory exists (which has the CF customization) it no longer freaks out.
That’s it. I’m running Solr 1.4.1 on Windows 2008 R2 64-bit! This is so awesome since I didn’t need to make any other changes, such as JVM settings, and other tweaks, they all carried over!

WTH?! Bloglines to Shutdown!

Ugh. Bloglines, which I use for reading 1000+ blogs, is shutting down in 3 weeks. They say you can export the OPML file, but what about all my saved/marked content? This sucks.

Lessons Learned: Moving from Verity to Solr (Part 3)

Previously, in Part 2 of this series, I blogged about some difficulties in working with Solr. I am following up with some more lessons learned.
This one deals with wildcards. If you look on page 359 of the 2nd WACK book, it states: “A search for ?ar?et would find both Carpet and Target, but not Learjet.”
Thanks for Ray Camden for confirming this would actually NOT work. I believe he said it would go in the Errata. Why?
Well, starting with a wildcard value, either * (star) or ? (question mark) will fail with Solr. You will get this nice error: “Error executing query : orgapachelucenequeryParserParseException_Cannot_parse_XXX__or__not_allowed_as_first_character_in_WildcardQuery”.
So although it would nice to search for “*ing”, it would be impractical according to the Solr folks. Is there a way around this? Well, theoretically yes.
Lets say in column1 you wanted to search for ?ar?et just like in the example. Do the following:

1. When you build your SQL query, add a column and do a REVERSE. For example: SELECT column1, column2, REVERSE(column1) AS reverseColumn1..
2. Index the results.
3. Then when searching, do a reverse of the term if it starts with a wildcard. In this example: “te?ra?”