2600hz Mobile on the VUC
VoIP Users Conference
Tomorrow, our very own Darren Schreiber will be appearing on the VoIP Users Conference (that’s Friday the 28th) at noon EST (9am PST). This will be 2600Hz’s second appearance. Darren will be discussing the 2600hz Mobile rollout, among other things. We’ll be talking about getting mobile data speeds up to 60mbps (nope, that’s not a typo!) and connecting voice calls to your very own FreeSWITCH, Asterisk or other such server.
Tune in to catch a live demo (what could go wrong???) of provisioning a mobile phone onto the new 2600hz Mobile network!
More information here:
Erlang Factory Conference + Kazoo Training
Craving your Erlang fix? This is a call for those that would like to get some training on the Erlang backend of Kazoo! We have 10 attendees so far, but would obviously love to have you join us too! Learn all about how the backend works, how to extend various apps like Crossbar and Callflows, meet the Kazoo Erlang team (which has grown recently - welcome Luis!), and have a fun time doing it. If you can make Erlang Factory the week before, all the better!
More information here:
We’re excited to again be participating in Kamailio World in Berlin, Germany April 3rd and 4th. This awesome event lets us learn a ton about the super-scalable SIP border software we use, Kamailio, while hanging out with our awesome European friends. Please come join us and meet folks from our team as well as everyone else at Kamailio World!
More information here:
And Even More Trainings…!
We’re working on finalizing some dates for Basic and Advanced (new!) training courses for Kazoo for those who can’t make the Erlang Factory.
In addition, we’ve refactored our FreeSWITCH training and are excited to launch another series of great FreeSWITCH trainings to support that awesome conference.
Please check our training site, http://www.voipkb.com, for more details - we hope to have the rooms and dates secured next week.
As always, thanks for being a part of this project.
The 2600hz Team
High Performance Telecom, Section 1: Distributed BLF + Kamailio Module
TL;DR: We wrote a Kamailio module and we rewrote BLF from scratch. It’s legit.
To say our engineers have been busy would be an understatement. We’re all about building the biggest and baddest systems on the planet, and in order to do that we have to push the barrier of what is possible. We think you’ll be quite pleased with how we’ve implemented our manifold performance improvements, and if you’re like us, this is the kind of stuff that really gets your gears going. Let’s get to it!
Performance Kazoo Kamailio Module
One major bottleneck we were aware of in v2 was our sending of registrations to FreeSWITCH in addition to phone calls. The reality is that all registrations hit Kamailio first and then go to FreeSWITCH, duplicating the work that FreeSWITCH has to do. That’s why we hired Anca Vamanu - one of the authors of some Kamailio modules and an expert C coder who has been an absolutely fantastic add to our project.
Anca helped simplify the problem - why not have registrations go to Kamailio and then to AMQP for us to handle directly? After some intense design work with our CTO, Karl Anderson, the answer was clear - treat Kazoo as a database (but really, send database requests to AMQP to be handled any way we see fit). This strategy fit in perfectly with the rest of our architecture, which utilizes AMQP as a messaging engine for real-time or configuration data. And the strategy worked! The result of connecting Kamailio directly to AMQP (and thus, our WhApps) is a massive performance improvement.
This is the culmination of six months of day in, day out work. What we’ve figured out over the last few years is that certain components in the network serve multiple roles, but that, at scale, isolating functionality is actually the best way to move forward. Our revelation was this: FreeSWITCH is one of, if not the, best media server on the market, but Kamailio is an amazing SIP Signaling program. We have, thus, stripped FreeSWITCH down to just Core Media handling and moved all non-media related signaling to our SBC, Kamailio.
But wait, why?
To understand why we did all of this work, you have to understand the crux of the issues with registrations. Registrations do two things: 1) they tell us what IP address and port the phone is operating on, and 2) they open a pinhole in the firewall; creating a small hole in the network to let traffic outside of the corporate network in. Obviously it’s of paramount importance to minimize the ports that are open on the firewall to reduce the overall attack surface (more ports open is more places the bad guys can get in). Firewalls hate open ports, particularly ports and they are designed to close inactive ones. To solve this, we (and every other vendor out there) send bogus traffic to and from the phone all day long – giving the illusion that activity is occurring between the two endpoints and allowing the pinhole to remain open.
The problem for us is that we use multiple origination servers, so when a call is being processed by us, it may not necessarily be via the server it came from last time. That’s part of distributed systems engineering, and anchoring a particular customer to a particular box is not distribution. Let me be clear this is a hard problem to solve, but we gave it our best shot!
FreeSWITCH deals with registrations in a manner we found hard to distribute. Essentially FreeSWITCH writes every Registration to a local SQLite Database (Read: SLOW!!!). You can optionally upgrade that mechanism to use Postgres or MySQL, but now you are managing a database which we find difficult to utilize in a real-time, distributed cluster. Because we haven’t found a way to distribute the registrations within FreeSWITCH, we are writing to SQLite and then buffering to Memory (Read: SLOW!!! x2). We thought there had to be a better way.
Turn out, the creators of FreeSWITCH envisioned scenarios where registrations weren’t inside FreeSWITCH. Woohoo! So we decided to circumvent the registration process completely, but still have FreeSWITCH handle calls. The results were literally nothing short of amazing.
Our new solution completely removes FreeSWITCH from the equation and routes all registrations to our Kamailio Registrar Server. This doesn’t require us to echo any messages across servers, it’s simpler, with fewer transactions, lower bandwith, less latency AND the media server can be separated from the SBC (which we couldn’t do prior to this change). Our experience is that the bottleneck in most large production networks has been registrations, and by offloading the most painful part of our stack to something purpose-built to handle those difficulties we get numerous benefits. Beyond what I’ve mentioned already, we get two other massive consequential benefits:
- FreeSWITCH is less busy so it can process more calls
- Kamailio is WAY better at handling SIP Packets for signaling than FreeSWITCH so it uses less threads overall
When FreeSWITCH isn’t doing registrations, it can support more calls (more CPU Idle for the win!). Kamailio is really really fast; we’re seeing improvements greater than 25x over our previous infrastructure implementation. This will allow us to cut our infrastructure requirements for large carrier networks significantly. We’ve also redone other portions of our network leveraging Kamailio.
High-Performance BLF Rewrite
Out of the box, FreeSWITCH excels at single server installations with all phones attached to a single box. We run a lot more than 1 FreeSWITCH server, so we have to deal with shared state across all servers. This is an incredibly challenging issue and BLF represents one of the biggest pain points – any phone may be subscribed to any box and other phones may cause the state of that subscription to change. We’d like to eliminate these kinds of difficult transactions wherever we can. When we say expensive, you can basically read that as REALLY SLOW! Basically, our Erlang Call Manager (ecallmgr) has to query each FreeSWITCH server to see who has what phone registered and which BLF subscriptions are active. Since we run 2x ecallmanager, it actually queries each FreeSWITCH server twice. Across sometimes 20 servers. So in other words, one light changing status could be 20 requests across all servers, 18 of them being discarded.
That’s a ton of packets! Literally, in a 10 freeswitch server install with 2x ecallmanagers running, we’re sending 20 packets to identify where phones are reporting BLF. This is akin to an internal denial of service attack; there’s just a very large amount of packets transversing our internal networks, and we’d like to reduce our load if possible.
Our solution was to keep a small table in memory that shows where each phone is, thereby greatly reducing the transactional costs of identifying the location of a given handset. It turns out the memory utilization is quite low. By querying one shared table as opposed to numerous individual tables, we reach consensus in one query instead of 400. One can see the 400x performance boost right there, although in real world trials it tends to be closer to 25x, as previously mentioned. That table can also be sharded across multiple servers via a key, as needed, to allow infinite growth.
To review… Performance Part 1, we rewrote BLF and wrote a Kamailio Module from scratch. Both of these new implementations bring tremendous utility to our network in the form of massive speed improvements and capacity innovation. We think you’ll love this tech, and we hope you see our passion every day when you use our work.
Does High Performance Telecom make you salivate? Would operating a network with 1/10 of your current infrastructure be beneficial to your company? We think so. Email us at email@example.com to talk about how we can help your business, today!
A Very v3 February
To start the new year off right, we’ve got one more present leftover from the holidays. As you may have heard, we’ve been crazy busy building the third version of our software. Kazoo started out as an attempt to build a better communications system and, over time, it has become one of the best available.
Today, we are pleased to announce a streamlined installation experience with the launch of theKazoo v3 ISO and the Kazoo v3 installation tool!
We hope this allows some of you still running v2 to see all of the awesome benefits of v3. We’re happy to announce further improvements in our latest version. Most notably v3 includes:
- One command installation (and single server support!)
- The Most stable and Secure Kazoo so far
- Simply Massive Performance Improvements
- Much, Much more!…
To Use the new ISO
While a wiki page for the ISO is still pending, installation should be self-explanatory. Just mount the ISO and boot, following any on-screen instructions. You must still apply IPTables yourself to secure the system, so please tread carefully as we work out some of the kinks on this ISO. For now, it’s awesome for development, but not quite recommended for production.
To Use the All-in-One Installers Script
Please head on over to https://2600hz.atlassian.net/wiki/display/docs/Kazoo+Single+Server+Install which has detailed instructions on how to use the shell script to perform a single-server installation.
We hope you enjoy using these two new additions to the 2600hz Kazoo toolset and we hope you’ll consider providing feedback and contributions to the project to keep things rolling!
The 2600hz Team
1 2 3 4 5
Updates!! Orange + Erlang Factory
We are tremendously excited to announce that Orange, the French Telecom Operator, has selected us for a partnership program. It was an incredibly competitive process and from among hundreds of companies, Orange selected 2600hz and six others to participate.
Patrick Sullivan, our COO and Co-Founder, will be traveling to Paris next week to hang out with our lovely hosts. If you’re near Paris do let us know and we can try to set up a meeting!
Second, we’re honored to announce that one of our architects, James Aimonetti, will be presenting a training at Erlang Factory in March. This is an Erlang focused Kazoo training and we think it will be quite a fun day.
Here’s the registration link , make sure to select the option for the 3 day training or the 3 day training + conference.
See you soon!
"The Event was everything I was hoping for, and any business interested in a state of the art VoIP platform should be at KazooCon 2014." -Kevin Charbonneau, Netitek LLC
In case you missed it, a couple weeks ago we pulled off a minor miracle. Take my word for it, completing your first conference is nothing short of magic. It takes a lot of planning, elbow-grease and a heck of a lot of luck to make something magical happen, but when everything lines up it is beautiful.
This is the story of how we pulled off our first conference: KazooCon.
"KazooCon: Getting you paid. Great Conference" -Jose Paz, Quik.is
KazooCon Behind the Scenes: Planning
Let’s start with the first thing we had to do – come up with a plan for KazooCon.
The first step to planning a conference is deciding you want to throw a conference. This is usually not done on a whim and, frankly, internally there was a ton of discussion about the timing and the ability of our team to perform while simultaneously handling multiple large projects. The first things we agreed upon were:
- Small, intimate audience (100 attendees)
- Non-traditional Venue
- Elegant and Illustrative
We wanted the venue to be small and intimate for a few reasons, but, to my mind, the ones that stand out were these two. First, we didn’t know how many tickets we would be able to sell. At first, no one was buying tickets, but as the ticket offers began to expire we saw a ton of people grabbing the tickets (we actually sold out almost two months in advance!!). Second, we are a bootstrapped company, so throwing $20,000 on a venue was out of the question. This leads into the non-traditional venue. By leveraging our knowledge of San Francisco, we were able to find a venue that was not only gorgeous, but affordable because it’s not normally a conference venue (if you’re thinking of throwing an event in SF, we can’t recommend The Grand enough. The staff was amazingly helpful and the venue is great). To say our choice of venue worked out would be an understatement.
Lastly, 2600hz wants to bring elegant telecom to the masses. That is our goal, and our conference should reflect that. In addition, we are absolutely obsessed with learning, and, having been to many other telecom conferences over the years, we think our audience wants to learn too. At KazooCon, we wanted the experience to feel more authentic than traditional conferences and we took steps to make that real.
One final question: Why should you hold a conference? In our opinion, now was the right time because we have so much technology launching and because we felt there was a chance we could actually pull it off. There was definitely a chance in the beginning that KazooCon wasn’t going to come together, but as time went by there was no question the event was going to happen. We are incredibly proud to have pulled it off, now let me tell you a bit more about how we did it.
“KazooCon was a real-time demo of what 2600hz is made of, innovative people with a revolutionary telecom solution. I can’t wait to see more about the Mobile VoIP/MVNO solutions“ -Michael Dury
What worked onsite?
First and foremost, food is really important. If you provide bad food at your conference or don’t provide refreshments, you’ve failed. Keeping your attendees fed and happy is of critical importance, so make sure you provide hearty food that people can enjoy while they learn. We used the catering service from Herbs and Spices and it was great! We had breakfast + snacks throughout the day and provided lunch as well as appetizers during the happy hour.
We leveraged two tiers of seating and a distributed seating arrangement which gave our attendees the freedom to move about the conference hall even during presentations without disturbing other guests or presenters. Our architecture is responsive and fault tolerant, so we thought our conference should be too. On the ground floor we had the stage right up front, the audience a few steps from it and refreshments/food in the back. Upstairs we had all of the sponsors and our engineering team setup in a corner. The beauty of this arrangement was that when people had questions, they could walk upstairs and get an answer immediately. This worked out great for everyone: our attendees got great answers, the sponsor booths had amazing foot traffic and our team got to engage all of the people in the auditorium.
We also covered the walls and floors with Telecom memorabilia including handsets from days gone by and infrastructure components from many networks long forgotten. Yes, we’re nerds, but we love it and we hope you do too :D. We stand on the shoulders of giants and we want to share that world with you.
Since we had a whole exhibit hall full of engineers, we knew that the Wifi experience had to be solid gold. Originally, we had intended on running two 100mbps links from Comcast and two separate discrete networks (one for the audience and one for operations). We had intended to use two meraki routers, two meraki 48port switches and 5 ubiquiti access points. In reality, this was way overkill. We ended up using one of the circuits, one of the routers, one of the switches and 2 of the access points. Ask anyone at the event, the wifi was killer. The key components to a successful experience with Wifi are this: don’t overcomplicate things and make sure you have 250kbps per person. Our aggregate capacity for the event was approximately 2mbps per person and it was overkill. Thanks to Felix from SFTelco for graciously donating the routing equipment to our event and to Mahesh from Ubiquiti Networks for the Access Points.
A quick note about event bandwidth: it is hard to obtain. Most of the quotes we got for the event were 5k+ for 10mbps, which just didn’t make sense to us. We’re telecom people and a lot of us have been slinging circuits since the frame relay days; paying $5,000 for a 2 day ethernet drop just doesn’t make sense. If you can get it, cable operators tend to be the cheapest option, and you can get a lot of bang for the buck. We got 200mbps for $1200 and it was amazing. Thanks to Brad from Comcast for hooking us up.
"This is the best Telecom conference I’ve ever been to" -L.H.
Ain’t No Party like an Afterparty
So, we threw a little shindig after KazooCon. Most conference shindigs tend to feel sort of plastic, and we just don’t get down like that. Our plan was simple: human interaction. How do we drive human interaction? With compelling games and activities.
For our evening, we gave each person who attended the afterparty a letter to hang around their neck and, using scrabble rules, we had the attendees make words. The better the word, the fancier the drink; we even had multiplier rules for words that were related to Telecom or 2600hz. Basically, it was a smashing success. I’m still getting emails from people telling me about the words they spelled and the people they met.
Key takeaway: Make it fun! We’re all adults but the best networking events are the ones where everyone has an excuse to talk to everyone. Make an excuse for your attendees to talk to each other that isn’t work related and the world becomes an oyster.
"KazooCon was a very special blend of the industry’s most innovative people, leveraging the industry’s most innovative platform to change the telecom landscape forever. I can’t wait for next year." -Ron Shoe, SIPOasis
Things we learned for next time
Speakers will flake and you should have a backup presentation ready to go. Demos will fail and you should always make a canned version to failover to in the event of a live catastrophe. If you don’t get at least one request that makes you say “WTF” you aren’t running a serious conference. Don’t panic, everything can be solved with enough time and energy.
Folks like theatrical displays of intelligence, but they don’t like being preached to; make sure you walk the line without stepping over. If not everyone in your audience is technical, you have to be careful of the mix of technical to non-technical discussions. Some people will love the tech stuff and some don’t, but the technical people will definitely get bored if you omit all of the technology. If you have the scale, multiple tracks is definitely the way to go, but we didn’t think that was an option the first year.
Thank You and a few thoughts on 2014
KazooCon was epic and we can’t wait to do it again. Next year we plan to make a few changes:
- Bigger Venue (250 capacity)
- Multiple Tracks
- More Demos!! <—- More interactivity too
- Less schwag and more art
Conferences should be an experience that provokes thought and provides motivation for the hard year ahead. We want KazooCon to be the place where telecom folks get inspired to build great things. Here’s hoping you can make it to the next one, we’d love for you to join us next October in San Francisco.
Join us at KazooCon 2014!!
"The beautiful thing about the Kazoo platform is the fact that you don’t need a deep understanding of telephony to build rich applications using their complete REST APIs. 2600hz Mobile Integration: The missing link in the VoIP revolution." James Lyons, QuikVoip
Thanks again to all of our terrific sponsors; without you this would not be possible. There are also a ton of photos on Facebook here.
P.S. Content recaps from KazooCon are coming!!! Stay tuned to blog.2600hz.com for more information!
Amdahl’s Law and Parallel Computing
We talked a lot about the various computing power laws in a previous blog post, but one of the themes was the ascent of parallelization in code. We argued previously that the shift in power to performance ratios, as opposed to pure power, will result in non-parallel code producing diminishing returns against the potential of Moore’s Law. Put simply, we think that if your code isn’t parallel, you’ll be working much harder than your competitors and achieving far less.
What is Parallel code?
In its simplest definition, parallel code is code which scales linearly as you add additional cores. Obviously, there are practical limits, for example, super computers almost always require tons of custom code refactoring in order to see line-rated performance, but what we’re talking about is taking advantage of the shift to pooled resources, or Cloud Computing. Whereas previously one had to “rack ‘n stack” their own equipment, the ability to spin up (or down) servers on command has changed the industry. Perhaps more importantly, because beefier, bigger machines are more expensive to manage, consuming lots of cheap computing becomes more important. Here’s the difference Amazon makes in a Nutshell.
Previously: If I had 1000 Photos to process, and it would take me one hour per photo on my home computer, that’s it, I’m probably either going to modify that procedure and do something less intensive or I’m going to forget the idea all together, because I can’t wait 1000 hours.
Total Time: 1000 Hours
Now: If I have 1000 photos to process, and it would take me one hour per photo on my home computer, I rent 1000 Amazon instances the same size of my computer and run 1 photo on each of them for an hour.
Total Time: 1 Hour
It’s totally impractical for me to own and maintain 1000 servers at my house or even nearby, and yet, with Amazon, I can harness the power of those servers on-demand to solve problems that would’ve been too cumbersome to tackle before. Whether I rent 1 computer for 1000 hours or 1000 computers for 1 hour, the cost is the same.
So that’s great! I can save time, you can save time, we can all work a good bit faster because of AWS. For something simple like the photo processing, this is pretty obvious, but, unless you’re doing something pretty trivial, chances are you’ll have parts of your application you can serialize in this manner, and parts of your application that you can’t serialize. As luck would have it, this phenomena is described in an awesome theorem entitled “Amdahl’s Law”.
The limit of parallelization is the non-parallel portion of the program. -Amdahl’s Law
Essentially, Amdahl’s law is saying that applications are always limited by the portion of the calls that can’t be serialized. If you have an application that takes 3 minutes to run and 50% of the code could be parallelized and 50% can’t, even if you reduce the parallel section to all operate simultaneously, the upper bound will always be 90 seconds. In short, there are some parts of an application which just have to run before you can address serialization points. The key to winning in the future is to placate Amdahl’s law as much as possible, ergo, to write code that is can be parallelized, from top to bottom. Whatever is in your app that can’t be parallelized will become the bottleneck, because, inevitably, it’s the only part of the program that can’t take advantage of Moore’s law.
The limitation of parallel code is non-parallel code. Often, they’re used in tandem. It might very well be impossible to make code that can be run completely parallel, but the closer you can get, the more horsepower your application will be able to consume over time. Parallelization is a generational gap in computing, only this time it’s not hardware making the leap, it’s your code.
At 2600hz we wrestle with code constantly. Finding points of serialization is of paramount importance, and we were recently able to parallelize a large number of API requests which dramatically improved the experience of using our GUI and our APIs. Over time, these points of parallelization will only prove more valuable. At the same time, we think about delays just as much. “Why does this program have to wait for that program to respond?” or “Is there a way we can do this before that?”. When you write parallel code, you have to think about eliminating gaps, and eliminating interruptions (like global garbage collection under load). Erlang, with it’s per-actor garbage collection model, excels at enabling management of individual processes even under extreme load. For us, operating a globally distributed cluster of Telecom systems with a shared messaging bus would be impossible without Parallel code, and in our offices this manifests in our love for Erlang.
In summary, Parallel code is the recipe for unlocking Moore’s Law. Amdahl’s law says that the slowest part of your app is the non-parallel portion. Figuring out how to make more things run at the same time is really important, and will only increase in importance over time. At 2600hz, we want to be able to “spin ‘n scale” and, for us, we see parallel code as the only way to achieve that at the pace of Moore’s Law.f
2600hz Mobile: The Announcement
In case you missed it, this week at TechCrunch Disrupt, we announced that we are now a mobile virtual network operator (MVNO). 2600hz is utilizing one of the nation’s top cellular networks to do this (Sprint). This means we have a physical connection to Sprint’s network and permission to utilize their mobile network for carrying our voice and data services. This is in addition to the existing functionalities we already offer.
What exactly are we connected to?
- Provisioning Systems
- Voice Billing Feeds
- Data Billing Feeds
- Voice Services
- SMS Services
- Data Services
- Porting Systems
- Tier 2 Support Services
- Core Switching Network
- Cellular Tower and Network Status Information
In other words:
- Your phone calls automatically go from the cell phone, to the cell tower, to us. (Both in and out)
- Your requests to activate, manage, restrict or change service plans on phones go to us, then to the cell towers via automation
- Your billing automatically goes from the cell network to us, via automation.
- Even geo-location (via triangulation of cell towers) can be accessed automatically, via APIs.
And we take all this cool stuff and we pass it on to you, via simple APIs you can integrate.
We’ll be releasing these things initially at our first-ever conference, KazooCon ( http://www.kazoocon.com/ ) but in the meantime we’ve been getting a lot of questions from curious folk about what all this is about. So I’m going to answer that as best I can here.
Our announcement about mobile this week is a really big deal, but we suspect most people don’t get why. That’s OK. My excitement is very high, but in order to understand why, I think you need some history and context, so let’s start there…
Why I Am Interested In Telecom
Since I was a kid, telecom has always been a black box that I had wanted to crack open. It was a technology that connected everyone, was relied on by everyone, but was understood by very few. It was like the lights in a broadway show – if they don’t go on, the show can’t happen – but if they work flawlessly, nobody seems to care. Telecom, to me, is the same - it’s the behind-the–scenes, “how the world works” mystery of technologies that everyone utilizes.
My adventures with telecom got serious around age 13. I would take weekend trips to the public library, where I would checkout every telecom-related book I could find (at one point, the library restricted me from checking out so many books). I kept a box of dismantled phones in my room, where most kids would keep toys. I could identify people’s voicemail passwords just by listening to the touchtones they entered. I used to break the phone lines in our house, just to see the telephone repairman come over so I could watch how he worked (my parents were forever convinced that “the rain” kept making our phones go out…). For some bizarre reason, I was hooked.
As I grew up, I began to resent how restricted all this cool technology was. At some point, there were no more books to read. I was bored with the physical parts of the phone. I knew there was this whole big network out there I was plugging into with my phone. But I could never get access to the network. I couldn’t explore what was making that dial tone. I couldn’t learn how billing or switching worked. How is it that, when the power goes out, the phone lines keep working? That stuff was all secret. It was frustrating, to say the least.
When I was 15, I got my first look inside a real telco. I got to tour the phone company in my town! And I finally started to get some answers. How did the phone keep working during a storm, when the power was out? Simple! Batteries! Whole rooms full of them. I got to see where all the wires in the town terminated into a big switch, literally the physical manifestation of modern switchboards. I even learned that orders you placed for service changes, like adding call-waiting and Caller-ID, were sent to the local switching office by - you won’t believe this - dot matrix printer! The order would be taken at the telco’s call center and entered into the billing system. Then it would be sent to the local telco office’s printer, and finally, taken by an engineer at the telco and manually entered into the switch itself. How’s that for archaic? That’s originally why it took 24 hours for the service to become “active” and why they charged an activation fee! (Later, they automated all that, but still kept charging the fees) Silly stuff, but still, it was fascinating. So this is how the telco worked!
How 2600hz Came to Be
Fast-forward to age 18, when I got my first telco-job, and also found Asterisk, the first major piece of open-source telephony software. I’ll never forget the first time I played with Asterisk: I did not sleep. I stayed up the entire night trying to figure out what in the heck FXO was and why my phone wouldn’t ring. I bought the Asterisk book and read it everywhere I went. When I finished it, I read it again. It didn’t matter. It was this entirely new world where I could control my own destiny. I could simulate million-dollar features with just a few lines of code. It was awesome.
Not before long, I had setup some real Asterisk installations and found it’s limitations. It worked OK, but it had it’s fair share of problems. It certainly wasn’t easy. But I had a real-world use for my long-time hobby! Then I stumbled onto FreeSWITCH. FreeSWITCH was amazing. It was clearly a step-up from everything that came before. It was faster, it had more levels of modularity and sophistication, and it integrated with everything. Clearly, this was the future, but in a different way. These were hardcore toys now, the tools of professionals. Talk was around hundreds of calls per second, thousands of calls per box, etc. No more small office PBX, this was big. I dove in head-first to FreeSWITCH.
At the same time, SIP Trunks started popping up everywhere, for cheap. This wasn’t just cheap calling - the information inside the SIP packets was genuine. You could see if a call was from a pay phone, or had been forwarded, or manipulate Caller ID freely, all for less than a penny a minute. You could even see which carrier the call really originated from. This was access to the real network. Where the big boys play. Talk of a few phones was silly - talk of millions of phones was relevant. Here was this complicated, worldwide technology that, finally, I could explore and be a part of. This was where I wanted to be.
And as I got more involved in FreeSWITCH, I started learning about other people who wanted to be a part of this network, too. And I also learned where they kept struggling. All these different parts I’d been learning about for years were really complicated for new folks to understand and learn. It simply took time. And all the pieces you need in order to be able to offer those services to customers? Well… they require work to put together in a bundle that an average consumer can understand. That’s when I realized that maybe we could help make this easier to do.
That’s when 2600hz was born.
The Goal of 2600hz
The goal behind 2600hz has always been fairly simple. Make access to all this cool technology available as a simpler bundle, for others to use, sell or integrate. Make it easier to get access to this telco technology, in pieces or as a whole, and figure out ways to do more with it. Our strategy was also simple - listen to people’s repeat complaints and problems, the things people were trying to focus on the most, and help remove or resolve those issues while, at the same time, removing the general barriers of entry into the market itself. We’re still not even close to done, but we’re most definitely heading in the right direction.
So why am I telling you all this?
Frankly, I arrived late to the game of VoIP and SIP. Minutes were already a commodity. Good network-level switching software was expensive, but it was already available (if you could afford it). Our initial goal was to tackle the low-hanging fruit - make the switches themselves less expensive, easier to setup and simpler to use. That was the major opportunity 2600hz set out to explore. Part of that was catching up to features others were already offering.
Kazoo v3 (which we’ll talk about in a few weeks) is a significant milestone in that endeavor. We have finally reached the point where we feel our switch is mature. Sure, it still has a few rough edges, but it’s reliable, it works, it’s become really fast and it’s become proven. Which leads to the question - if that was the “low hanging fruit”, what’s next?
In parallel to us building a switch, we were also aware that the telco industry is changing - and rapidly. The focus has gone mobile. People want to take their data, their voice, their SMS messages everywhere they go. I’m not just talking mobile phones, though they remain the most obvious method people utilize to go mobile. I’m talking merging cellular networks with internet based technologies, landline services, even satellite networks - true world-wide access to telco features no matter what network you were on.
And that’s our real goal: Network unification.
But until this week, mobile was a huge missing component. That all changed on Monday.
Mobile Networks: Back to the Black Box
Why is our announcement about mobile significant?
I believe most things in life go in cycles. Mainframes went to desktops, and now with the cloud we’re essentially going back to mainframes. Each iteration is different - we learn something new each time and the environment is constantly changing - but the cycles themselves seem to repeat. Politics, technology, even your own life - you start as a kid needing someone to do everything for you, then you become self-sufficient, only to get old and need someone to take care of you again. Life, it seems, is an infinite loop. A circle.
The future of telecom looks eerily similar.
Landline networks went from hardcore black-boxes and secret codes to open books and simpler software. Go ahead - Google “SS7” or “wireline communications”, and everything is out there now. Almost nothing is secret. It wasn’t that way 20 years ago, though.
Now, try that with mobile? Back to black box we go. If you’ve paid attention to mobile, the cell carriers have worked very hard to keep much of their technology secret, patented and inaccessible. The FCC seems to be ok with this, making acquisition of wireless spectrum almost impossible for any new entrant. The mobile carriers like it this way - they’re in a fierce war with each other and they don’t seem to want new entrants meddling with all the money and time they’ve invested in their technology. This is an understandable business model - cell towers aren’t cheap to operate! - but back to my original problem listed above, it’s frustrating. Network access is again gone, restricted, unavailable.
And that’s why our announcement this month is so significant. To my knowledge, we’re the first company offering true, open access to the mobile network’s inner workings. We have done the dirty work of contracts, MPLS hookups, acquiring hardware, BGP routing, more contracts, setting up billing systems, permissions, rates, porting procedures and more. We’ve then tried to figure out how to bridge the gap between “complicated mobile stuff” and “simple VoIP stuff.” It took us nearly a year to do all this. And we’re finally arriving at our destination. Or at the starting line, I should say.
But our goal is different from other companies. We’re not in this to just sell the most $20/month cell phone plans.
Our end-goal : making it so you can figure out neat things to do with this technology. You can sell it, build a brand, or create something unique. We want to give you access to the cellular network.
Our integration isn’t so much significant because we’re offering cell service. It’s significant because we’re the first company to basically try to open-source cellular integration. We’re trying to encourage the hackers, the tinkerers, the enthusiasts out there in VoIP to get engaged in the next generation of networks - cellular. We’re trying to give you a way where you don’t have to spend $250,000 to just be able to play around with a test cell phone on your own little mobile network. We’re trying to get you thinking, “gee, what could I do that the cell companies can’t on their own today?” Or “gee, I’d love to solve this little problem that only impacts a small amount of people, but I can’t afford to run a cellular mobile network - is there an alternate option?”
Now the answer to those questions is yes, you can go ahead and build your cellular dream. And you don’t need unreliable over-the-top VoIP stacks running on cell phones to be able to do it.
So, that’s what 2600hz Mobile is all about. Opening up this network to you.
Where Do We Go From Here?
Yesterday, I personally worked with one of our awesome operations engineers (Jeremy) in our datacenter to put in place the last physical connection cable. This cable physically ties together Sprint’s network with ours. Watching the link light turn on and getting an email from the Sprint engineer that he saw it, too, was totally awesome. We’re really hooked up!
So now, we move on to beta testing. For the first few weeks, we’ll use the network ourselves within our office walls. Then, at KazooCon, we’ll share some cell phones and some early-access APIs with our KazooCon attendees. You’ll be able to activate some test cell phones we’ll raffle off at the conference (assuming all goes well next week). We’ll elicit feedback from you all.
And then come the ideas. We’ll open up the network to another group of participants, and we’ll ask you to start building your own ideas.
You’ll think of an idea. You’ll activate a cell phone. And you’ll try to put the idea in place. Then figure out how to market it, and give it a shot. If you don’t succeed, we’ll encourage you to try another idea. Until you find one that fits. Then, off you go on your cellular adventure.
What kind of ideas am I talking about?
Things I Could Think Of Doing With This…
The list below represents ideas that I personally could think of. Obviously you can use these yourselves, but my hope is you’ll come up with even more ideas.
If you’re a software enthusiast, there are some cool integrations you could do…
- You could make it so every time someone hits a form on a website, your sales team is alerted via SMS and can call the prospect from their corporate caller ID
- You could integrate cell phones with our upcoming call center app and give your agents cell phones to use. The cell phones could charge the agent for personal calls but you could pickup the tab on the call center calls (all automatically, via APIs, CDRs or Pivot)
- You could integrate cell phone activation with your own website, where you also activate VoIP phones, and cell VoIP + cellular bundles together
- You could integrate cell phone VoIP stacks and native cellular access to create your on cellular + WiFi hybrid service
If you’re a VoIP enthusiast…
- You could assign international numbers to cell phones, so that people traveling from overseas can activate a US cell phone with a US number that also comes with an international number. Relatives in their home country would dial a local number to reach them, while US folks would also dial a local number to reach them. All with the same cell phone
- You could assign an international Caller-ID to a cell phone, even though it’s in the US. This way, someone traveling can keep their home number when making calls
- You could assign two phone numbers to the same cell phone, and use a prefix to indicate which Caller ID to display on all outgoing calls
- You could record all calls made on both the cell phone and the desk phone, both in and out of the cell phone, or only record them when a prefix is dialed
- You could transcribe calls while they’re in progress, from both mobile or desk phone
If you just like making money…
- You can sell bundles that integrate cellular and VoIP desk phones for less than the cost of most cell phone plans
- You can sell packages to businesses that include phones for everyone, on shared plans
For the advanced players…
- You can setup an MPLS link between a customer’s private network (hospital, government, etc.) and route calls from IP phones at their site over a secure, dedicated connection to the cellular network - never touching the internet
- You can allow in-progress calls to be transferred seamlessly between cell phones and desk phones with the push of a BLF button
Or… <insert idea here>
We know this is all new, so we’ll be doing seminars, trainings and all sorts of other events to teach you how to build your ideas.
Now, It’s Your Turn. Come Join Us.
At the end of the day, the reality is this that setting all this up took a lot of work. Work that’s not easy to replicate. And our engineers expect to eat, sleep, have somewhere to live, etc. so it also takes money.
This is the same for all companies. That’s why most companies who do this integration work think of their own products and try to sell them, without giving access to you. It’s much less risky as a business prospect.
But I’m driven by something different. I think the world is changing. I think the demand for access to the inner workings of the network is outpacing the demand that a single company can keep up with in telecom. I think people who follow 2600hz have a lot in common with me. I think all of you want to be a part of telecom technology, but you want to build something unique. I think all of you want to innovate.
It is this belief that led us, at 2600hz, to figure out how to crack-open the mobile network and see what you do with it. We want to give you that access.
Honestly, I am hoping you all get a little bit of joy out of this project, even if you never launch a real product. When you hit the SEND button on the cell phone in your house, and see a SIP packet come in to your home PBX, I hope you find the same little bit of joy I found when I first tried out Asterisk.
We hope we’re right in our mission. I personally can’t wait to see what you do with this new-found telecom freedom.
Thanks for reading all this. I hope it gives some insight into what we’re up to.
In the coming weeks, we’ll be releasing more details, beta sign-ups and the list of APis we’ll be supporting (once we’ve tested them). This all will come out some time after KazooCon. If you can’t wait, then we recommend you join our conference for early-access and the ability to play with the technology sooner. You can sign-up and view the schedule here: http://www.kazoocon.com/
To be explicit: Attendees of KazooCon get first access to 2600hz mobile. KazooCon is the only place you can talk to the 2600hz team about 2600hz Mobile and the best place to learn about our Kazoo Platform. Join us in San Francisco, October 14th and 15th.
Get your tickets here: http://kazoocon.eventbrite.com
Computing Over Time
When we think about advances in computing power, most people think of Moore’s Law, Kryder’s Law, or Nielsen’s law, which are about the pace of technical development. Moore’s law is the most well known, and talks about the pace of computational intensity doubling every 18-24 months. Kryder’s law is about storage, and doesn’t have a speed attached, but notes that storage capacity increases at a much faster pace than Moore’s law. Finally, we have Nielsen’s law, which is about Network throughput. Essentially, Nielsen theorized that since Network performance was increasing slower than computational intensity, we would always be network-bound (ergo the network would always be the limitation) and for all intents and purposes this has proved to be pretty much true.
So we have the three immortal axioms of computing: processors double all the time, storage moves faster and networks move slower. That’s all well and good except for one thing…
Moore’s law is broken
In case you didn’t get the Memo, Intel hit the power wall in 2005. Basically, up until about 2005, Intel was just making one chip as fast as possible. Damn the torpedoes, Intel was going to make the biggest and baddest chips on the planet. One day they woke up and companies like Facebook were buying AMD processors because the performance to power ratio was so much better. In fact, the cost of operating Intel’s Beefy Single-Core chips was such that many companies were evaluating other computing platforms.
How were these companies beating Intel, the juggernaut of CPU systems? More cores. So in 2005, Moore’s law was really fractured, at least from the users perspective. Why? Computer programs were almost universally written for one core because Parallel computing is hard. It’s getting easier, but it’s still hard.
The companies that will dominate in the future are those that can consume many cores simultaneously for processing, versus the companies who dominated in the past simply by throwing more processors at the problem. You can still throw money at the problem, but your code has to change or you’ll suffer diminishing returns on your investment. That’s why Moore’s law ostensibly broke in 2005, because code doesn’t automatically scale with hardware anymore.
Yes, Moore’s law still holds true doubling speed every 18-24 months, but now you do so by adding ever-more cores. Right now, the programming languages many people use are not built to take up ever-more cores, or do so but with a lot of expensive overhead to make this transparent to the programmer and the user. So the limitation now is less infrastructure and more creativity (programs written for multi-core environments will now run exponentially faster than clumsily written ones).
Amazon Web Services and Computing Over Time
AWS, coincidentally, was founded in 2006, launched by Jeff Bezos as an experiment in aggregated computing. The thesis was this: sell computing like a utility. Jeff and company benefited from a lot of trends, but none was more important than the shift in importance of power to performance ratios. Amazon wanted cheap chips that didn’t use a lot of power, and they wanted a lot of them. What many of us don’t realize is what a large component of the overall computing ecosystem electricity becomes at scale. Power, is everything.
AWS, like most Amazon services, is playing the long game. Jeff’s plan is pretty simple: If Amazon operates at 0 margin for a long enough time, no one will be able to compete. It’s an interesting idea, but one that can really only be proven over time. Which brings me to my idea of a new Law: Bezos’ Law.
The Cost of Cloud Computing will be cut in half every 18 months - Bezos’ Law
Like Moore’s law, Bezos’ Law is about exponential improvement over time. If you look at AWS history, they drop prices constantly. In 2013 alone they’ve already had 9 price drops. The difference; however, between Bezos’ and Moore’s law is this: Bezos’ law is the first law that isn’t anchored in technical innovation. Rather, Bezos’ law is anchored in confidence and market dynamics, and will only hold true so long as Amazon is not the aggregate dominant force in Cloud Computing (50%+ market share). Monopolies don’t cut prices.
So where does that leave us? I suppose that depends on how much you love infrastructure. If you love it, this is probably irrelevant to you, but if you don’t this means that the cost of your application will be cut in half every year and a half. That’s a big deal.
Conclusions and/or Considerations
If you’re a programmer, and if you believe in these predictions and the approaches people are taking to improved capacity and performance, your programming style may need to evolve with these changes, too. In short, write Parallel code because it’s better, and realize that the world is migrating towards more cores as quickly as they can because of power consumption patterns and operational considerations.
A Quick Note
At 2600hz, we think a guy named Joe Armstrong nailed it when he wrote the language Erlang. There’s a lot to love, but some of the key highlights for us are trivial parallelization, actor-model with per-actor garbage collection, and massive serialization. We use Erlang as the core of our stack, binding together our Border Controller, Media Servers, Databases, and Applications. It’s not always about the language you use but in our use case, we’ve found Erlang to be scalable, modular, and ultimately fit to the task of managing Global Communications Infrastructures. We feel that this tech, along with the engineers implementing it, give us a significant edge in getting to market and staying online. In addition, it’s a rough world out there and in short order companies that can’t parallelize will be left with massive legacy cost structures.
In a nutshell, Parallel code is where it’s at and at 2600hz we leverage Erlang to help get us to many cores. We’re well on the way and we think this is the direction computing is heading. Parallel Computing ensures Moore’s Laws longevity.
Do you think parallel code is the right answer? Are you worried about your legacy communications system not scaling? Talk to us, we can help. Ring us at firstname.lastname@example.org to chat today!
2600hz Expert Q&A Recap
Whew, the first part of this year has really gone quickly. We want to take some time out to reflect on the stuff we’ve discussed over the last few weeks and to talk about where the Expert Q&A is going from here.
The first Q&A session we did was on Virtualization. This is a huge topic, but we focused on Virtual Machine Timing because that has such a big impact on communications applications. Lots of great stuff in this Q&A featuring our friends from Voxeo and PSSC Labs.
The second Q&A session we did was on Faxing. Again, another huge topic but we decided to go after Faxing because it impacts almost all of our clients. What was really interesting to me about this was the idea that the delimiter in faxing is silence, as opposed to most IP protocols where signaling comes from packets. It’s very clever and I encourage you to dive into this presentation, it’s Jam Packed!
The third Q&A session we did was on Provisioning. We think the industry can do better when it comes to provisioning. There isn’t one provisioner that handles all handsets, but there should be, and that’s what we’re working on at 2600hz. This Q&A presentation explains why this is such a hard thing to do and how our team is solving this problem. This presentation features Andrew Nagy of Provisioner.net fame!
The third Q&A session we did was on Database. Could we have picked a more massive topic? Maybe, but we decided to focus critically on Layer 1 Failures and Network Partitions, which are some of the biggest issues operationally for distributed systems. Featuring Sam Bisbee from Cloudant, we dive headfirst into a TON of content on Databases. This one is not to be missed, it’s a great deck and a ton of awesome commentary. (The image at the top of this email is about Zombie, Flapping Datacenters, which have to be removed from the cluster).
Where to now?
First up we’ve got two more Awesome Q&A sessions scheduled. The one Next Friday, the 14th of June is on Session Border Controllers! If you can’t tell the difference between an SBC and a Firewall, this is the presentation for you.
Two weeks after that we’re covering DTMF! Ever wonder where those tones on your phone came from? We’re going to give you the answers.
If you want to resell 2600hz, we have two great trainings coming up. The first one is this Friday, the 7th of June and the second one is two weeks later on Friday, the 21st of June.
Last but not least, our sales team just got a new shiny 800 #. Ring them at:
Thanks, and we look forward to bringing you tons of new content. Be on the lookout for an announcement of our API Trainings which should be starting later this month or early next!
How Squirrels break Datacenters and other Database Conjectures
This Q&A presentation was influenced by Kyle Kingsbury’s work on Jepsen, an exploration of modern databases. If you haven’t seen his work and you like this stuff, you should go check it out. It’s awesome.
We just did our Epic Database Expert Q&A featuring Sam Bisbee of Cloudant and Darren Schreiber of 2600hz. We covered a range of topics but focused on these three kinds of failures:
- Network Partitions
- Layer 1 Disasters
- Flapping Internet (Special Class of Network Partitions)
All public networks are unreliable; such is the reality of modern distributed database management (and let’s face it, because of AWS we’re all managing distributed databases, whether we like it or not). Sometimes, when these unreliable networks break down, a partition can form. These partitions, depending on your database configuration, can wreak havoc across a wide gamut of scenarios.
Arguably, most of what a database admin does is prepare for network partitions and how to resolve them.
-Joshua Goldbard, 2600hz
Yes, modern databases run fairly well when they’re not in a failure state, but, frankly, the only thing that matters is the failure state. During a partition, it’s important to understand your database behavior, which can vary wildly. At 2600hz, we leverage BigCouch which is a Master-Master replication strategy with Dynamo Quorum(PPT LINK). What that means in plain English is that every node is a master node and it uses consistent hashing to redistribute the load in the event of a partition.
The best advice we can give here is to know the failure modes and behavior of your database and understand the partition realities of the software.
-Darren Schreiber, 2600hz
Layer 1 Disasters
Hurricanes, Earthquakes or Squirrels? Squirrels eating glass. Squirrels caught in HVAC units. Squirrels tampering with Power lines. All of these are examples of Layer 1 Disasters, but we only think about the really massive outages, not the unexpected ones that effect critical infrastructure.
Darren, the 2600hz CEO, has a lot of experience managing Datacenters. Here’s a quick story from back-in-the-day about managing racks in a DC:
Once upon a time a Datacenter vendor decided to give my company a couple of months notice that they were going to 10x our rates. They assumed we couldn’t migrate out of that Datacenter easily, and they were right. Because we were cheap, we did everything ourselves, which meant loading the racks into a pickup truck by hand that we drove in the rain to another Datacenter. Not my definition of Fun.
-Darren Schreiber, 2600hz
Contrast that with our experience during Sandy, when we were using BigCouch:
On the day before the storm, we just turned off the Datacenter. That was it.
We can evade storms, earthquakes and Squirrels because of Cloudant.
-Darren Schreiber, 2600hz
If a Datacenter gets into a Layer 1 issue, we just kill it and move on. When the disaster is mitigated we bring the service back up, but losing an entire DC (or even multiiple DCs) is not an issue because of our database choice.
Protip: If you can’t predict disasters, have a plan to avoid them.
It is up or is it down? Flapping internet is a special case of the Network Partition. Basically, a flapping connection is one that goes down, then comes up, then goes down; this is actually worse than a server going hard down because of the reconciliation process that happens when the networks reconnect. We’ve got one answer for this and one only: Zombie Servers get Double Tapped.
Basically, if a DataCenter is flapping, it’s better to just disconnect that datacenter manually until it can be confirmed as restored. There’s no easy way to say this, Flapping is one of those scenarios that requires manual intervention. If the DC is flapping you have to take it out or you may never get back online.
Protip: If it flaps, Double Tap.
Darren chose to use the last few minutes to pontificate about how ridiculous life was before BigCouch. There was a point very early on where we simply could not get BigCouch to work and we thought we might have to fold the company. Thanks to incredible support from the Cloudant team, we got everything working and the rest is history.
It’s night and day. We just don’t spend any time on the database anymore… We just don’t have problems with the Software.
-Darren Schreiber, 2600hz
Sam chose to talk about right reliability, specifically in the way in which other systems buffer writes and respond to concerns about right availability.
There are a lot of other databases out there that will reply “Write confirmed” when you buffer the write, NOT when it actually commits to disk. The practical effect of this is that if the disk dies before the write moves out of the buffer, you’re missing writes, which is death to a database.
Durable databases write to disk and confirm, they don’t just buffer. Databases that buffer can be very dangerous depending on the workload.
-Sam Bisbee, Cloudant
We had a blast doing this presentation with Cloudant and we can’t wait for the next Q&A Session on Border Controllers in two weeks. Click here to join us!
Two weeks after that, we’re going to discuss DTMF and how all of that nonsense works in VoIP. Register for free here:
Lastly, if you’d like to talk to our friends at Cloudant, check them out at Cloudant.com or in IRC on Freenode in channel #Cloudant.
Thanks so much for checking out our Q&A. If this all sounds like it’s too much work, you should call our Sales team at 8554642600 or email@example.com. We power some of the biggest infrastructures on the planet and we’d love to talk about how we can help your business eliminate the pains of operating communications infrastructure :).