Web Software Architecture and Engineering – Life on the Bleeding Edge

Archive for August, 2010

Cumulative Hotfix 1 (CHF1) for ColdFusion 9.0.1

Now available! More details @ http://kb2.adobe.com/cps/862/cpsid_86263.html.

Real-world check for Is IP Valid using ColdFusion. 4 Techniques Examined.

As you may know, recently I wrote a cgi facade. In that facade, we check for certain headers, and prefer them over CGI.REMOTE_ADDR, which may be incorrect if a proxy is involved.
Well, I assumed that the proxies configured to send those headers would be doing so correctly. Turns out, one of our new clients, and unnamed government entity was passing bogus data using Squid. Did I mention I hate Squid?
Anyways, so after quickly looking at the bad data, I decided to add a simple check to see if the data is a valid IP (IPv4). Lo and behold, what I found out there was inconsistent in validating for IPv4.
I started at CFLib, where I found the IsIP UDF written by Nathan Dintenfass.
Second, I found a CF Function called validIpAddress written by Wil Genovese. It uses Regex to find a valid IP.
Third, I found a undocumented, in-built CF method that supposedly validates IPs, in a blog post by Anuj Gakhar.
And last, Joseph Lamoree, my co-worker, looked up an old UDF he had written called isIPV4.
I ran a series of tests against all 4, and here is what I found:
IP: 0.1.1.1
Check using isIP UDF: true
Check using validIPAddress: true
Check using Internal CF:
* validateIPAdress: YES
* validateIPv4Address: YES
* validateIPv6Address: NO
Using Joseph’s UDF: false
IP: 0.0.0.0
Check using isIP UDF: false
Check using validIPAddress: false
Check using Internal CF:
* validateIPAdress: YES
* validateIPv4Address: YES
* validateIPv6Address: NO
Using Joseph’s UDF: false
IP: 255.255.255.255
Check using isIP UDF: false
Check using validIPAddress: false
Check using Internal CF:
* validateIPAdress: YES
* validateIPv4Address: YES
* validateIPv6Address: NO
Using Joseph’s UDF: false
IP: 192.168.0.0
Check using isIP UDF: true
Check using validIPAddress: false
Check using Internal CF:
* validateIPAdress: YES
* validateIPv4Address: YES
* validateIPv6Address: NO
Using Joseph’s UDF: false
And finally, IP: 127.0.0.1
Check using isIP UDF: true
Check using validIPAddress: false
Check using Internal CF:
* validateIPAdress: YES
* validateIPv4Address: YES
* validateIPv6Address: NO
Using Joseph’s UDF: true
As you can see, Joseph’s simple UDF really shined, and the Internal CF methods acted weird. The other functions were inconsistent.
I can understand the difference between validation according to standards set by a RFC and looking for real-world IP. I was surprised there isn’t someone who had posted something rock-solid before for ColdFusion without getting heavy into Java like CFDNS.
Here is the code I used: http://pastebin.com/TgnSG7iL.

Git is Scary – But it doesn’t need to be!

I watched two Git preso over the last week by CF folks. They both tried really hard to make Git sound better than SVN, and easier to use.
But looking at their preso, if I was a newbie to source code management, I’d be scared. All that Git bash stuff is a nightmare!
Git isn’t perfect, nor is it well suited for web development. But it also doesn’t need to be scary.
As a long time SVN (and previously CVS) user, I moved to Git for all development earlier this year, but this was after watching preso and researching Git for years! And I found one common thread, it does better merging, but it can also be scary. Why? There aren’t any good tools!
Earlier this year, the great folks at syntevo GmbH launched SmartGit, and it made Git sooooo much easier to use and manage. This is what made the switch from SVN (for which they have a killer tool called SmartSVN) so much easier, and really did it for me.
Maybe someday someone will create a version control tools best suited for web development, without the confusing terminology that Git and SVN has, but for now, we’ve found a great combo that works.
Each developer has his own branch in which he works, then we have a dev branch, a staging branch, a UAT branch, and a prod/master branch. Developers commit code in their branch on a daily basis. When they feel ready they merge in (which is very easy with SmartGit) their changes to the Dev branch. As other developers have also been committing code to the dev branch, they also merge out their changes from dev back into their branch, so that they have the latest changes. The dev branch is where intergrations testing can occur, all developers code meets up, and we do our own testing on the dev box. Once we’re good, we merge from dev to stage, where QA does their testing on the staging server. And then when tickets are passed, then we merge from stage to UAT, which is on the same environment externally as prod. If we need to show off any code to customers early, we can do that here, and we also make sure the code runs OK in prod. And finally, when code has passed UAT testing, then we merge to MASTER, and push the update to the servers on the next launch. Works well for us.
Developers may also create their own experimental branches on the fly. I have one right now for testing our move from Verity to Solr, for example.
The only problem with this model is that you have to be careful when pushing up, as you can push up code that hasn’t been fully passed. So you can cherry pick commits, or just be careful. This is where the “art” of source control comes in, there is no exact science, its all what works best for your team.
My greatest wish for source control would be if you could tag a commit as being at a specific level only. So if you commit code, and say this is, for example, “dev” only, any merge from dev to stage would ignore that code, until the ticket associated with that code passes on dev, and then it’ll move up. This would make merges soooo much easier, and you could be sure your code will never make it to prod unless its passed at that level by QA. Anyways, wishful thinking.
Oh, and Git integration into Eclipse is half-baked, so much so its useless for me. I always have SmartGit and CFBuilder open side by side. Good thing is that SmartGit runs on most platforms!

Need Help : SQL Injection Tools

QA is working on hacking our system running the full gamut of security concerns (without costing much money). We’re not looking to hiring an outside firm, just yet.
And one of the things we are having a hard time finding is a tool that test for SQL Injection. Most of what we’ve found is overly complex or simply outdated.
Does anyone know of any tool – simple and easy to use, that can run through a site attempting all sorts of sql injection attacks?

Intel buying McAfee!

Hot news! Chipmaker’s formal foray into Enterprise software.
Read more @ http://finance.yahoo.com/news/Intel-buying-McAfee-in-push-apf-2695002564.html?x=0.

DNSMadeEasy DDOS Attack

Today, I received a letter from the President of DNSMadeEasy. I use their service, and have found it to be very powerful and far cheaper than their competitors. At first I was shocked to read they suffered a DDOS attack of the scale of 50GB/s. They have an illustrious 8 year 100% uptime that is now marred. However the letter I got was so refreshing and honest, I was amazed. See below (if this sort of thing interests you!).
*****
Dear DNS Made Easy Client,

On August 07, 2010 DNS Made Easy was the target of a large multi Gb/s attack against all of our name servers.   The attack started at 8:00 UTC and was fully mitigated by 14:00 UTC.  During this time period there were regional outages from some or all of our name servers.  Regional outages means that certain regions of the world were not able to resolve your DNS and other regions of the world were resolving normally.  When all name servers were not reachable a DNS query would have been lost, when some name servers were not reachable then DNS performance would have been slower than normal but still operational.

The regional downtime was in very small periods but it still did affect the overall resolution for all of our client’s DNS.  It is for this reason that we are explaining the situation in full to all of our clients now.

1) How long were the DNS outages?
In some regions there were no issues, in other regions  outages lasted a few minutes, while in other regions there were sporadic (up and down) outages for a couple of hours.  In Europe for instance there was never any downtime.  In Asia downtime continued longer than other regions. In United States the west coast was hit much harder and experienced issues longer than the central and east coast.

2) Many clients have asked us if in fact there was downtime since they did not notice issues.
Many clients did not notice any DNS downtime.   In fact many clients would not have noticed this issue if we had not sent this email.  But we feel disclosure of this issue is something that we owe our client base. 
If you want to see if there is a significant loss of DNS queries you can quickly compare your daily queries from this Saturday to last Saturday in the DNS Made Easy control panel.  Overall query statistics comparing this Saturday’s query load (minus attack traffic) to recent Saturdays’ query loads shows that our servers properly responded to a query total this Saturday within a 2% difference from recent Saturdays.

3) Where did the attack come from?
We believe that the DDoS came from a botnet attack originating from Asia.  Most attack traffic originated in or transited through China.  The source IPs appear to be mostly spoofed but the vast majority are assigned by APNIC to Chinese Networks and Chinese ISPs.  Traffic levels reported to us by our bandwidth providers regarding their connections through which this traffic entered their networks also points to origins in Asia.

4) How large of an attack was this?
This attack hit levels that were so high that our Tier1 upstreams were suffering latency and network issues for other clients at many of their locations due to this attack.  This caused some of our Tier1 bandwidth providers to use their last resort response of null routing traffic to some of our IPs from some networks to prevent major service degradation to their core networks. 
Measuring the exact size of this attack is rather difficult.  However, discussions with our Tier1 bandwidth providers during the attack led to an estimate of 50 Gb/s in size.  This was based on reports of multiple 10Gb/s lines being saturated at multiple different providers in different geographic regions.
During our after-action discussions internally and with our providers after the attack was mitigated we analyzed all information available to us through monitoring systems and traffic reports and we revised our estimate of the attack size to be fluctuating between 20Gb/s and 40Gb/s during the attack.  We will never know the true size of this attack as we actively moved traffic around to different locations throughout the attack and IPs were temporarily null routed into and through various networks, and some traffic was blocked from provider to provider in response to the attack.
We do know that due to the service implication to the Tier1 providers, networking teams from China Netcom, China Telecom,  Level3, GlobalCrossing, Tiscali, and Arbinet were involved to stop the attacks.  Level3 and Arbinet both played special heroic roles in facilitating that the correct people were involved from all networks to make sure that the attack was stopped as quickly as possible.

5) How was this attack stopped?
Fighting attacks of this magnitude is very complex and a full answer involves much information that we do not want these criminals to know.  What we can say is that that we used a combination of routing techniques, DDoS mitigation tools, customized firewalls, and high level inter-provider negotiations.
China Netcom and China Telecom had to null route the name servers from their networks in order for the attack to not impact other traffic they had going to the United States. 

6) Will an SLA credit be issued?
Yes it will be.  With thousands paying companies we obviously do not want every organization to submit an SLA form.  Even though not all clients noticed the attack, we plan on issuing an SLA to every single paying DNS account.
You will be receiving an email about the SLA credit to your account in the next few days. 

7) Does this affect your 100% uptime history?
Yes, any service outage would result in loss of uptime.  We had a history leading uptime of over 8 years of 100% uptime.  With a calculated two hour outage (which is probably longer than we were actually down for anyone) this DDOS attack put our overall uptime history at a calculated 99.9999%.  This is still an excellent uptime history.

8) What would it take to get your 100% uptime history back?
That is mathematically impossible.  But we can work on increasing our 99.9999% uptime history and we will work hard on building another run of more than 8 years of 100% uptime.  We are confident that we can do it and we look forward to the challenge.

9) Would another DNS provider have been able to stop this attack?
We are sure that our competitors will claim that the answer is yes.  In fact we have been called by several of our competitors with very amusing phone calls during and after the attack asking us to update our website to say that we no longer have a 100% uptime history (which we have started and will complete soon).  This was a very large attack, so we do not believe that other DNS services could have stopped it either.  If any of our customers are considering leaving our services based on this issue, then we would recommend highly that you request a detailed report for how any new potential DNS provider would deal with an attack of this magnitude.  Please note that this was our first issue of downtime over our 8+ years of providing enterprise managed DNS services.

10) What is the next step?
At this time all DNS resolution is functioning as intended from all of our global locations.
In our 8+ year history, we have had numerous attacks against our services.  Historically we have been able to mitigate these attacks without any service degradation. One thing we have always taken away from every attack is a deeper understanding of what we need to do to make our network and services stronger and more reliable.
This DDoS attack against us was different from others in that the size was massive enough that our standard mitigation strategies were not sufficient to prevent several network nodes from being flooded.  We now have a deeper understanding of what happened during the attack and have started planning network upgrades and mitigation strategies to help fight these criminals in the future.  It is, and always has been, our commitment to make the DNS Made Easy network the strongest and most reliable DNS network in the world.

11) Can I pay more for a higher level of service with DNS Made Easy?
We believe that we provide more service per dollar than any competitor in the DNS industry.  This is why we have the best ROI in the industry.  We do not do this by cutting networking cost.   As many of
you aware DNS Made Easy feels we can cut costs by eliminating a lot of the sales (including commissions), presales, and unnecessary marketing expenditures.
Everyone at DNS Made Easy feels that our network is as strong as or stronger than any competitor in the United States and Europe and you can verify this with speed tests and our highest industry uptime.   As all DNS Made Easy customers know, as our customer base grows, so does our network.  This is how we can continually keep adding to our network and always remain a fraction of the price of our competition.
You will hear more from our network team as we plan on adding additional precautions to keep everything running smoothly during attacks in the future.

One thing that I want to say is that we sincerely apologize that this happened to your DNS service.  We understand that hundreds of thousands of domains rely on our DNS services each day to keep their businesses running smoothly.  This is not something that we treat lightly and this is not something that we are going to just let slip away.  We have already started to plan on building a network to focus on preventing attacks like this from causing any service disruption in the future.
Everyone here at DNS Made Easy would like to thank you for your continued loyalty and kind words during this time.  We can easily say the DNS Made Easy customers are the best in the business.

Question, comments, concerns?

Please let us know.  I personally will be answering as many tickets and questions as possible in the following weeks.  Our full DNS Made Easy staff is dedicated to answering your questions and easing any concerns that you have.

Regards,
-Steven Job
President and Founder of DNS Made Easy

CFCollection Documentation Incorrect

For action=”list” it states:
“list: returns a query result set, named from the name attribute value, of the attributes of the collections that are registered by ColdFusion. If you have Solr and Verity collections and omit the engine attribute, ColdFusion lists information for both types of collections.”
The last statement is not true. If you omit the engine attribute, verity is the default. There is currently no way to get a complete list of collections using cfcollection.

Standalone Solr 9.0.1 Woes

I’ve been working with Solr a lot more lately. Its worked fine locally. I decided to try a standalone installation on our dev server so that all developers can access the same collection. However it seems the Solr installer isn’t as robust as Verity’s.
One issue is that in CFAdmin, Data & Services > Solr Server, you can type in any text or IP, and there is no indication as to whether CF was able to connect at all. It always returns a positive message: “Solr Server Configuration information updated.”. Either way, I typed the FQDN of the dev server.
I then went to the Data & Services > ColdFusion Collections page, and it cleared out my old Solr collections, but there was also no error here – it would be nice to know if it wasn’t able to connect!
So I tried the URL, http://<FQDN&gt;:8983/solr/, and it didn’t connect, so I supposed it was a firewall issue. So I hopped on the dev server, which is a Windows 2008 R2 OS, and went to the firewall area. No entry for Solr!
I suppose that since Verity uses specific ports like 9953, and that its not a web server returning data, its not as big a security risk. You’d have to know the specifics on the proprietary protocol. But, since Jetty is returning data on port 8983, and its a web server, this is where you need some security. It would have been nice though to see CF Solr as an entry in the firewall unchecked, and all you had to do was enable access to your subnet (domain) for the standalone to be complete.
To me, the standalone installer is locked into the server, unless you open it up. And the installer doesn’t do any of the work for you, nor does it notify you that you may need to make the following changes for standalone to work properly. Sigh.
Anyways, so I added inbound and outbound entries for the firewall and I am still unable to connect. Does anyone have any clues on any additional setup that is needed for a standalone Solr to accept remote connections? I am able to run the Solr web interface locally on the dev server just fine, just not from my workstation, and the firewall isn’t the issue, because I tried turning the firewall off completely and still no-go.
UPDATE: Looks like Jetty is set to block all NON-LOCAL connections. This makes the standalone installer pretty useless. We went to coldfusionsolretcjetty.xml, and changed -Set name=”Host”-127.0.0.1-/Set- to -Set name=”Host”-0.0.0.0-/Set- (substitute dashes with <>). But now its open to everyone, and ideally you’d like to enter just your subnet or set of IPs. Looks like I have to drive into Jetty now!

Follow

Get every new post delivered to your Inbox.

Join 413 other followers