"The Event was everything I was hoping for, and any business interested in a state of the art VoIP platform should be at KazooCon 2014." -Kevin Charbonneau, Netitek LLC
In case you missed it, a couple weeks ago we pulled off a minor miracle. Take my word for it, completing your first conference is nothing short of magic. It takes a lot of planning, elbow-grease and a heck of a lot of luck to make something magical happen, but when everything lines up it is beautiful.
This is the story of how we pulled off our first conference: KazooCon.
"KazooCon: Getting you paid. Great Conference" -Jose Paz, Quik.is
KazooCon Behind the Scenes: Planning
Let’s start with the first thing we had to do – come up with a plan for KazooCon.
The first step to planning a conference is deciding you want to throw a conference. This is usually not done on a whim and, frankly, internally there was a ton of discussion about the timing and the ability of our team to perform while simultaneously handling multiple large projects. The first things we agreed upon were:
- Small, intimate audience (100 attendees)
- Non-traditional Venue
- Elegant and Illustrative
We wanted the venue to be small and intimate for a few reasons, but, to my mind, the ones that stand out were these two. First, we didn’t know how many tickets we would be able to sell. At first, no one was buying tickets, but as the ticket offers began to expire we saw a ton of people grabbing the tickets (we actually sold out almost two months in advance!!). Second, we are a bootstrapped company, so throwing $20,000 on a venue was out of the question. This leads into the non-traditional venue. By leveraging our knowledge of San Francisco, we were able to find a venue that was not only gorgeous, but affordable because it’s not normally a conference venue (if you’re thinking of throwing an event in SF, we can’t recommend The Grand enough. The staff was amazingly helpful and the venue is great). To say our choice of venue worked out would be an understatement.
Lastly, 2600hz wants to bring elegant telecom to the masses. That is our goal, and our conference should reflect that. In addition, we are absolutely obsessed with learning, and, having been to many other telecom conferences over the years, we think our audience wants to learn too. At KazooCon, we wanted the experience to feel more authentic than traditional conferences and we took steps to make that real.
One final question: Why should you hold a conference? In our opinion, now was the right time because we have so much technology launching and because we felt there was a chance we could actually pull it off. There was definitely a chance in the beginning that KazooCon wasn’t going to come together, but as time went by there was no question the event was going to happen. We are incredibly proud to have pulled it off, now let me tell you a bit more about how we did it.
“KazooCon was a real-time demo of what 2600hz is made of, innovative people with a revolutionary telecom solution. I can’t wait to see more about the Mobile VoIP/MVNO solutions“ -Michael Dury
What worked onsite?
First and foremost, food is really important. If you provide bad food at your conference or don’t provide refreshments, you’ve failed. Keeping your attendees fed and happy is of critical importance, so make sure you provide hearty food that people can enjoy while they learn. We used the catering service from Herbs and Spices and it was great! We had breakfast + snacks throughout the day and provided lunch as well as appetizers during the happy hour.
We leveraged two tiers of seating and a distributed seating arrangement which gave our attendees the freedom to move about the conference hall even during presentations without disturbing other guests or presenters. Our architecture is responsive and fault tolerant, so we thought our conference should be too. On the ground floor we had the stage right up front, the audience a few steps from it and refreshments/food in the back. Upstairs we had all of the sponsors and our engineering team setup in a corner. The beauty of this arrangement was that when people had questions, they could walk upstairs and get an answer immediately. This worked out great for everyone: our attendees got great answers, the sponsor booths had amazing foot traffic and our team got to engage all of the people in the auditorium.
We also covered the walls and floors with Telecom memorabilia including handsets from days gone by and infrastructure components from many networks long forgotten. Yes, we’re nerds, but we love it and we hope you do too :D. We stand on the shoulders of giants and we want to share that world with you.
Since we had a whole exhibit hall full of engineers, we knew that the Wifi experience had to be solid gold. Originally, we had intended on running two 100mbps links from Comcast and two separate discrete networks (one for the audience and one for operations). We had intended to use two meraki routers, two meraki 48port switches and 5 ubiquiti access points. In reality, this was way overkill. We ended up using one of the circuits, one of the routers, one of the switches and 2 of the access points. Ask anyone at the event, the wifi was killer. The key components to a successful experience with Wifi are this: don’t overcomplicate things and make sure you have 250kbps per person. Our aggregate capacity for the event was approximately 2mbps per person and it was overkill. Thanks to Felix from SFTelco for graciously donating the routing equipment to our event and to Mahesh from Ubiquiti Networks for the Access Points.
A quick note about event bandwidth: it is hard to obtain. Most of the quotes we got for the event were 5k+ for 10mbps, which just didn’t make sense to us. We’re telecom people and a lot of us have been slinging circuits since the frame relay days; paying $5,000 for a 2 day ethernet drop just doesn’t make sense. If you can get it, cable operators tend to be the cheapest option, and you can get a lot of bang for the buck. We got 200mbps for $1200 and it was amazing. Thanks to Brad from Comcast for hooking us up.
"This is the best Telecom conference I’ve ever been to" -L.H.
Ain’t No Party like an Afterparty
So, we threw a little shindig after KazooCon. Most conference shindigs tend to feel sort of plastic, and we just don’t get down like that. Our plan was simple: human interaction. How do we drive human interaction? With compelling games and activities.
For our evening, we gave each person who attended the afterparty a letter to hang around their neck and, using scrabble rules, we had the attendees make words. The better the word, the fancier the drink; we even had multiplier rules for words that were related to Telecom or 2600hz. Basically, it was a smashing success. I’m still getting emails from people telling me about the words they spelled and the people they met.
Key takeaway: Make it fun! We’re all adults but the best networking events are the ones where everyone has an excuse to talk to everyone. Make an excuse for your attendees to talk to each other that isn’t work related and the world becomes an oyster.
"KazooCon was a very special blend of the industry’s most innovative people, leveraging the industry’s most innovative platform to change the telecom landscape forever. I can’t wait for next year." -Ron Shoe, SIPOasis
Things we learned for next time
Speakers will flake and you should have a backup presentation ready to go. Demos will fail and you should always make a canned version to failover to in the event of a live catastrophe. If you don’t get at least one request that makes you say “WTF” you aren’t running a serious conference. Don’t panic, everything can be solved with enough time and energy.
Folks like theatrical displays of intelligence, but they don’t like being preached to; make sure you walk the line without stepping over. If not everyone in your audience is technical, you have to be careful of the mix of technical to non-technical discussions. Some people will love the tech stuff and some don’t, but the technical people will definitely get bored if you omit all of the technology. If you have the scale, multiple tracks is definitely the way to go, but we didn’t think that was an option the first year.
Thank You and a few thoughts on 2014
KazooCon was epic and we can’t wait to do it again. Next year we plan to make a few changes:
- Bigger Venue (250 capacity)
- Multiple Tracks
- More Demos!! <—- More interactivity too
- Less schwag and more art
Conferences should be an experience that provokes thought and provides motivation for the hard year ahead. We want KazooCon to be the place where telecom folks get inspired to build great things. Here’s hoping you can make it to the next one, we’d love for you to join us next October in San Francisco.
Join us at KazooCon 2014!!
"The beautiful thing about the Kazoo platform is the fact that you don’t need a deep understanding of telephony to build rich applications using their complete REST APIs. 2600hz Mobile Integration: The missing link in the VoIP revolution." James Lyons, QuikVoip
Thanks again to all of our terrific sponsors; without you this would not be possible. There are also a ton of photos on Facebook here.
P.S. Content recaps from KazooCon are coming!!! Stay tuned to blog.2600hz.com for more information!
Amdahl’s Law and Parallel Computing
We talked a lot about the various computing power laws in a previous blog post, but one of the themes was the ascent of parallelization in code. We argued previously that the shift in power to performance ratios, as opposed to pure power, will result in non-parallel code producing diminishing returns against the potential of Moore’s Law. Put simply, we think that if your code isn’t parallel, you’ll be working much harder than your competitors and achieving far less.
What is Parallel code?
In its simplest definition, parallel code is code which scales linearly as you add additional cores. Obviously, there are practical limits, for example, super computers almost always require tons of custom code refactoring in order to see line-rated performance, but what we’re talking about is taking advantage of the shift to pooled resources, or Cloud Computing. Whereas previously one had to “rack ‘n stack” their own equipment, the ability to spin up (or down) servers on command has changed the industry. Perhaps more importantly, because beefier, bigger machines are more expensive to manage, consuming lots of cheap computing becomes more important. Here’s the difference Amazon makes in a Nutshell.
Previously: If I had 1000 Photos to process, and it would take me one hour per photo on my home computer, that’s it, I’m probably either going to modify that procedure and do something less intensive or I’m going to forget the idea all together, because I can’t wait 1000 hours.
Total Time: 1000 Hours
Now: If I have 1000 photos to process, and it would take me one hour per photo on my home computer, I rent 1000 Amazon instances the same size of my computer and run 1 photo on each of them for an hour.
Total Time: 1 Hour
It’s totally impractical for me to own and maintain 1000 servers at my house or even nearby, and yet, with Amazon, I can harness the power of those servers on-demand to solve problems that would’ve been too cumbersome to tackle before. Whether I rent 1 computer for 1000 hours or 1000 computers for 1 hour, the cost is the same.
So that’s great! I can save time, you can save time, we can all work a good bit faster because of AWS. For something simple like the photo processing, this is pretty obvious, but, unless you’re doing something pretty trivial, chances are you’ll have parts of your application you can serialize in this manner, and parts of your application that you can’t serialize. As luck would have it, this phenomena is described in an awesome theorem entitled “Amdahl’s Law”.
The limit of parallelization is the non-parallel portion of the program. -Amdahl’s Law
Essentially, Amdahl’s law is saying that applications are always limited by the portion of the calls that can’t be serialized. If you have an application that takes 3 minutes to run and 50% of the code could be parallelized and 50% can’t, even if you reduce the parallel section to all operate simultaneously, the upper bound will always be 90 seconds. In short, there are some parts of an application which just have to run before you can address serialization points. The key to winning in the future is to placate Amdahl’s law as much as possible, ergo, to write code that is can be parallelized, from top to bottom. Whatever is in your app that can’t be parallelized will become the bottleneck, because, inevitably, it’s the only part of the program that can’t take advantage of Moore’s law.
The limitation of parallel code is non-parallel code. Often, they’re used in tandem. It might very well be impossible to make code that can be run completely parallel, but the closer you can get, the more horsepower your application will be able to consume over time. Parallelization is a generational gap in computing, only this time it’s not hardware making the leap, it’s your code.
At 2600hz we wrestle with code constantly. Finding points of serialization is of paramount importance, and we were recently able to parallelize a large number of API requests which dramatically improved the experience of using our GUI and our APIs. Over time, these points of parallelization will only prove more valuable. At the same time, we think about delays just as much. “Why does this program have to wait for that program to respond?” or “Is there a way we can do this before that?”. When you write parallel code, you have to think about eliminating gaps, and eliminating interruptions (like global garbage collection under load). Erlang, with it’s per-actor garbage collection model, excels at enabling management of individual processes even under extreme load. For us, operating a globally distributed cluster of Telecom systems with a shared messaging bus would be impossible without Parallel code, and in our offices this manifests in our love for Erlang.
In summary, Parallel code is the recipe for unlocking Moore’s Law. Amdahl’s law says that the slowest part of your app is the non-parallel portion. Figuring out how to make more things run at the same time is really important, and will only increase in importance over time. At 2600hz, we want to be able to “spin ‘n scale” and, for us, we see parallel code as the only way to achieve that at the pace of Moore’s Law.f
2600hz Mobile: The Announcement
In case you missed it, this week at TechCrunch Disrupt, we announced that we are now a mobile virtual network operator (MVNO). 2600hz is utilizing one of the nation’s top cellular networks to do this (Sprint). This means we have a physical connection to Sprint’s network and permission to utilize their mobile network for carrying our voice and data services. This is in addition to the existing functionalities we already offer.
What exactly are we connected to?
- Provisioning Systems
- Voice Billing Feeds
- Data Billing Feeds
- Voice Services
- SMS Services
- Data Services
- Porting Systems
- Tier 2 Support Services
- Core Switching Network
- Cellular Tower and Network Status Information
In other words:
- Your phone calls automatically go from the cell phone, to the cell tower, to us. (Both in and out)
- Your requests to activate, manage, restrict or change service plans on phones go to us, then to the cell towers via automation
- Your billing automatically goes from the cell network to us, via automation.
- Even geo-location (via triangulation of cell towers) can be accessed automatically, via APIs.
And we take all this cool stuff and we pass it on to you, via simple APIs you can integrate.
We’ll be releasing these things initially at our first-ever conference, KazooCon ( http://www.kazoocon.com/ ) but in the meantime we’ve been getting a lot of questions from curious folk about what all this is about. So I’m going to answer that as best I can here.
Our announcement about mobile this week is a really big deal, but we suspect most people don’t get why. That’s OK. My excitement is very high, but in order to understand why, I think you need some history and context, so let’s start there…
Why I Am Interested In Telecom
Since I was a kid, telecom has always been a black box that I had wanted to crack open. It was a technology that connected everyone, was relied on by everyone, but was understood by very few. It was like the lights in a broadway show – if they don’t go on, the show can’t happen – but if they work flawlessly, nobody seems to care. Telecom, to me, is the same - it’s the behind-the–scenes, “how the world works” mystery of technologies that everyone utilizes.
My adventures with telecom got serious around age 13. I would take weekend trips to the public library, where I would checkout every telecom-related book I could find (at one point, the library restricted me from checking out so many books). I kept a box of dismantled phones in my room, where most kids would keep toys. I could identify people’s voicemail passwords just by listening to the touchtones they entered. I used to break the phone lines in our house, just to see the telephone repairman come over so I could watch how he worked (my parents were forever convinced that “the rain” kept making our phones go out…). For some bizarre reason, I was hooked.
As I grew up, I began to resent how restricted all this cool technology was. At some point, there were no more books to read. I was bored with the physical parts of the phone. I knew there was this whole big network out there I was plugging into with my phone. But I could never get access to the network. I couldn’t explore what was making that dial tone. I couldn’t learn how billing or switching worked. How is it that, when the power goes out, the phone lines keep working? That stuff was all secret. It was frustrating, to say the least.
When I was 15, I got my first look inside a real telco. I got to tour the phone company in my town! And I finally started to get some answers. How did the phone keep working during a storm, when the power was out? Simple! Batteries! Whole rooms full of them. I got to see where all the wires in the town terminated into a big switch, literally the physical manifestation of modern switchboards. I even learned that orders you placed for service changes, like adding call-waiting and Caller-ID, were sent to the local switching office by - you won’t believe this - dot matrix printer! The order would be taken at the telco’s call center and entered into the billing system. Then it would be sent to the local telco office’s printer, and finally, taken by an engineer at the telco and manually entered into the switch itself. How’s that for archaic? That’s originally why it took 24 hours for the service to become “active” and why they charged an activation fee! (Later, they automated all that, but still kept charging the fees) Silly stuff, but still, it was fascinating. So this is how the telco worked!
How 2600hz Came to Be
Fast-forward to age 18, when I got my first telco-job, and also found Asterisk, the first major piece of open-source telephony software. I’ll never forget the first time I played with Asterisk: I did not sleep. I stayed up the entire night trying to figure out what in the heck FXO was and why my phone wouldn’t ring. I bought the Asterisk book and read it everywhere I went. When I finished it, I read it again. It didn’t matter. It was this entirely new world where I could control my own destiny. I could simulate million-dollar features with just a few lines of code. It was awesome.
Not before long, I had setup some real Asterisk installations and found it’s limitations. It worked OK, but it had it’s fair share of problems. It certainly wasn’t easy. But I had a real-world use for my long-time hobby! Then I stumbled onto FreeSWITCH. FreeSWITCH was amazing. It was clearly a step-up from everything that came before. It was faster, it had more levels of modularity and sophistication, and it integrated with everything. Clearly, this was the future, but in a different way. These were hardcore toys now, the tools of professionals. Talk was around hundreds of calls per second, thousands of calls per box, etc. No more small office PBX, this was big. I dove in head-first to FreeSWITCH.
At the same time, SIP Trunks started popping up everywhere, for cheap. This wasn’t just cheap calling - the information inside the SIP packets was genuine. You could see if a call was from a pay phone, or had been forwarded, or manipulate Caller ID freely, all for less than a penny a minute. You could even see which carrier the call really originated from. This was access to the real network. Where the big boys play. Talk of a few phones was silly - talk of millions of phones was relevant. Here was this complicated, worldwide technology that, finally, I could explore and be a part of. This was where I wanted to be.
And as I got more involved in FreeSWITCH, I started learning about other people who wanted to be a part of this network, too. And I also learned where they kept struggling. All these different parts I’d been learning about for years were really complicated for new folks to understand and learn. It simply took time. And all the pieces you need in order to be able to offer those services to customers? Well… they require work to put together in a bundle that an average consumer can understand. That’s when I realized that maybe we could help make this easier to do.
That’s when 2600hz was born.
The Goal of 2600hz
The goal behind 2600hz has always been fairly simple. Make access to all this cool technology available as a simpler bundle, for others to use, sell or integrate. Make it easier to get access to this telco technology, in pieces or as a whole, and figure out ways to do more with it. Our strategy was also simple - listen to people’s repeat complaints and problems, the things people were trying to focus on the most, and help remove or resolve those issues while, at the same time, removing the general barriers of entry into the market itself. We’re still not even close to done, but we’re most definitely heading in the right direction.
So why am I telling you all this?
Frankly, I arrived late to the game of VoIP and SIP. Minutes were already a commodity. Good network-level switching software was expensive, but it was already available (if you could afford it). Our initial goal was to tackle the low-hanging fruit - make the switches themselves less expensive, easier to setup and simpler to use. That was the major opportunity 2600hz set out to explore. Part of that was catching up to features others were already offering.
Kazoo v3 (which we’ll talk about in a few weeks) is a significant milestone in that endeavor. We have finally reached the point where we feel our switch is mature. Sure, it still has a few rough edges, but it’s reliable, it works, it’s become really fast and it’s become proven. Which leads to the question - if that was the “low hanging fruit”, what’s next?
In parallel to us building a switch, we were also aware that the telco industry is changing - and rapidly. The focus has gone mobile. People want to take their data, their voice, their SMS messages everywhere they go. I’m not just talking mobile phones, though they remain the most obvious method people utilize to go mobile. I’m talking merging cellular networks with internet based technologies, landline services, even satellite networks - true world-wide access to telco features no matter what network you were on.
And that’s our real goal: Network unification.
But until this week, mobile was a huge missing component. That all changed on Monday.
Mobile Networks: Back to the Black Box
Why is our announcement about mobile significant?
I believe most things in life go in cycles. Mainframes went to desktops, and now with the cloud we’re essentially going back to mainframes. Each iteration is different - we learn something new each time and the environment is constantly changing - but the cycles themselves seem to repeat. Politics, technology, even your own life - you start as a kid needing someone to do everything for you, then you become self-sufficient, only to get old and need someone to take care of you again. Life, it seems, is an infinite loop. A circle.
The future of telecom looks eerily similar.
Landline networks went from hardcore black-boxes and secret codes to open books and simpler software. Go ahead - Google “SS7” or “wireline communications”, and everything is out there now. Almost nothing is secret. It wasn’t that way 20 years ago, though.
Now, try that with mobile? Back to black box we go. If you’ve paid attention to mobile, the cell carriers have worked very hard to keep much of their technology secret, patented and inaccessible. The FCC seems to be ok with this, making acquisition of wireless spectrum almost impossible for any new entrant. The mobile carriers like it this way - they’re in a fierce war with each other and they don’t seem to want new entrants meddling with all the money and time they’ve invested in their technology. This is an understandable business model - cell towers aren’t cheap to operate! - but back to my original problem listed above, it’s frustrating. Network access is again gone, restricted, unavailable.
And that’s why our announcement this month is so significant. To my knowledge, we’re the first company offering true, open access to the mobile network’s inner workings. We have done the dirty work of contracts, MPLS hookups, acquiring hardware, BGP routing, more contracts, setting up billing systems, permissions, rates, porting procedures and more. We’ve then tried to figure out how to bridge the gap between “complicated mobile stuff” and “simple VoIP stuff.” It took us nearly a year to do all this. And we’re finally arriving at our destination. Or at the starting line, I should say.
But our goal is different from other companies. We’re not in this to just sell the most $20/month cell phone plans.
Our end-goal : making it so you can figure out neat things to do with this technology. You can sell it, build a brand, or create something unique. We want to give you access to the cellular network.
Our integration isn’t so much significant because we’re offering cell service. It’s significant because we’re the first company to basically try to open-source cellular integration. We’re trying to encourage the hackers, the tinkerers, the enthusiasts out there in VoIP to get engaged in the next generation of networks - cellular. We’re trying to give you a way where you don’t have to spend $250,000 to just be able to play around with a test cell phone on your own little mobile network. We’re trying to get you thinking, “gee, what could I do that the cell companies can’t on their own today?” Or “gee, I’d love to solve this little problem that only impacts a small amount of people, but I can’t afford to run a cellular mobile network - is there an alternate option?”
Now the answer to those questions is yes, you can go ahead and build your cellular dream. And you don’t need unreliable over-the-top VoIP stacks running on cell phones to be able to do it.
So, that’s what 2600hz Mobile is all about. Opening up this network to you.
Where Do We Go From Here?
Yesterday, I personally worked with one of our awesome operations engineers (Jeremy) in our datacenter to put in place the last physical connection cable. This cable physically ties together Sprint’s network with ours. Watching the link light turn on and getting an email from the Sprint engineer that he saw it, too, was totally awesome. We’re really hooked up!
So now, we move on to beta testing. For the first few weeks, we’ll use the network ourselves within our office walls. Then, at KazooCon, we’ll share some cell phones and some early-access APIs with our KazooCon attendees. You’ll be able to activate some test cell phones we’ll raffle off at the conference (assuming all goes well next week). We’ll elicit feedback from you all.
And then come the ideas. We’ll open up the network to another group of participants, and we’ll ask you to start building your own ideas.
You’ll think of an idea. You’ll activate a cell phone. And you’ll try to put the idea in place. Then figure out how to market it, and give it a shot. If you don’t succeed, we’ll encourage you to try another idea. Until you find one that fits. Then, off you go on your cellular adventure.
What kind of ideas am I talking about?
Things I Could Think Of Doing With This…
The list below represents ideas that I personally could think of. Obviously you can use these yourselves, but my hope is you’ll come up with even more ideas.
If you’re a software enthusiast, there are some cool integrations you could do…
- You could make it so every time someone hits a form on a website, your sales team is alerted via SMS and can call the prospect from their corporate caller ID
- You could integrate cell phones with our upcoming call center app and give your agents cell phones to use. The cell phones could charge the agent for personal calls but you could pickup the tab on the call center calls (all automatically, via APIs, CDRs or Pivot)
- You could integrate cell phone activation with your own website, where you also activate VoIP phones, and cell VoIP + cellular bundles together
- You could integrate cell phone VoIP stacks and native cellular access to create your on cellular + WiFi hybrid service
If you’re a VoIP enthusiast…
- You could assign international numbers to cell phones, so that people traveling from overseas can activate a US cell phone with a US number that also comes with an international number. Relatives in their home country would dial a local number to reach them, while US folks would also dial a local number to reach them. All with the same cell phone
- You could assign an international Caller-ID to a cell phone, even though it’s in the US. This way, someone traveling can keep their home number when making calls
- You could assign two phone numbers to the same cell phone, and use a prefix to indicate which Caller ID to display on all outgoing calls
- You could record all calls made on both the cell phone and the desk phone, both in and out of the cell phone, or only record them when a prefix is dialed
- You could transcribe calls while they’re in progress, from both mobile or desk phone
If you just like making money…
- You can sell bundles that integrate cellular and VoIP desk phones for less than the cost of most cell phone plans
- You can sell packages to businesses that include phones for everyone, on shared plans
For the advanced players…
- You can setup an MPLS link between a customer’s private network (hospital, government, etc.) and route calls from IP phones at their site over a secure, dedicated connection to the cellular network - never touching the internet
- You can allow in-progress calls to be transferred seamlessly between cell phones and desk phones with the push of a BLF button
Or… <insert idea here>
We know this is all new, so we’ll be doing seminars, trainings and all sorts of other events to teach you how to build your ideas.
Now, It’s Your Turn. Come Join Us.
At the end of the day, the reality is this that setting all this up took a lot of work. Work that’s not easy to replicate. And our engineers expect to eat, sleep, have somewhere to live, etc. so it also takes money.
This is the same for all companies. That’s why most companies who do this integration work think of their own products and try to sell them, without giving access to you. It’s much less risky as a business prospect.
But I’m driven by something different. I think the world is changing. I think the demand for access to the inner workings of the network is outpacing the demand that a single company can keep up with in telecom. I think people who follow 2600hz have a lot in common with me. I think all of you want to be a part of telecom technology, but you want to build something unique. I think all of you want to innovate.
It is this belief that led us, at 2600hz, to figure out how to crack-open the mobile network and see what you do with it. We want to give you that access.
Honestly, I am hoping you all get a little bit of joy out of this project, even if you never launch a real product. When you hit the SEND button on the cell phone in your house, and see a SIP packet come in to your home PBX, I hope you find the same little bit of joy I found when I first tried out Asterisk.
We hope we’re right in our mission. I personally can’t wait to see what you do with this new-found telecom freedom.
Thanks for reading all this. I hope it gives some insight into what we’re up to.
In the coming weeks, we’ll be releasing more details, beta sign-ups and the list of APis we’ll be supporting (once we’ve tested them). This all will come out some time after KazooCon. If you can’t wait, then we recommend you join our conference for early-access and the ability to play with the technology sooner. You can sign-up and view the schedule here: http://www.kazoocon.com/
To be explicit: Attendees of KazooCon get first access to 2600hz mobile. KazooCon is the only place you can talk to the 2600hz team about 2600hz Mobile and the best place to learn about our Kazoo Platform. Join us in San Francisco, October 14th and 15th.
Get your tickets here: http://kazoocon.eventbrite.com
Computing Over Time
When we think about advances in computing power, most people think of Moore’s Law, Kryder’s Law, or Nielsen’s law, which are about the pace of technical development. Moore’s law is the most well known, and talks about the pace of computational intensity doubling every 18-24 months. Kryder’s law is about storage, and doesn’t have a speed attached, but notes that storage capacity increases at a much faster pace than Moore’s law. Finally, we have Nielsen’s law, which is about Network throughput. Essentially, Nielsen theorized that since Network performance was increasing slower than computational intensity, we would always be network-bound (ergo the network would always be the limitation) and for all intents and purposes this has proved to be pretty much true.
So we have the three immortal axioms of computing: processors double all the time, storage moves faster and networks move slower. That’s all well and good except for one thing…
Moore’s law is broken
In case you didn’t get the Memo, Intel hit the power wall in 2005. Basically, up until about 2005, Intel was just making one chip as fast as possible. Damn the torpedoes, Intel was going to make the biggest and baddest chips on the planet. One day they woke up and companies like Facebook were buying AMD processors because the performance to power ratio was so much better. In fact, the cost of operating Intel’s Beefy Single-Core chips was such that many companies were evaluating other computing platforms.
How were these companies beating Intel, the juggernaut of CPU systems? More cores. So in 2005, Moore’s law was really fractured, at least from the users perspective. Why? Computer programs were almost universally written for one core because Parallel computing is hard. It’s getting easier, but it’s still hard.
The companies that will dominate in the future are those that can consume many cores simultaneously for processing, versus the companies who dominated in the past simply by throwing more processors at the problem. You can still throw money at the problem, but your code has to change or you’ll suffer diminishing returns on your investment. That’s why Moore’s law ostensibly broke in 2005, because code doesn’t automatically scale with hardware anymore.
Yes, Moore’s law still holds true doubling speed every 18-24 months, but now you do so by adding ever-more cores. Right now, the programming languages many people use are not built to take up ever-more cores, or do so but with a lot of expensive overhead to make this transparent to the programmer and the user. So the limitation now is less infrastructure and more creativity (programs written for multi-core environments will now run exponentially faster than clumsily written ones).
Amazon Web Services and Computing Over Time
AWS, coincidentally, was founded in 2006, launched by Jeff Bezos as an experiment in aggregated computing. The thesis was this: sell computing like a utility. Jeff and company benefited from a lot of trends, but none was more important than the shift in importance of power to performance ratios. Amazon wanted cheap chips that didn’t use a lot of power, and they wanted a lot of them. What many of us don’t realize is what a large component of the overall computing ecosystem electricity becomes at scale. Power, is everything.
AWS, like most Amazon services, is playing the long game. Jeff’s plan is pretty simple: If Amazon operates at 0 margin for a long enough time, no one will be able to compete. It’s an interesting idea, but one that can really only be proven over time. Which brings me to my idea of a new Law: Bezos’ Law.
The Cost of Cloud Computing will be cut in half every 18 months - Bezos’ Law
Like Moore’s law, Bezos’ Law is about exponential improvement over time. If you look at AWS history, they drop prices constantly. In 2013 alone they’ve already had 9 price drops. The difference; however, between Bezos’ and Moore’s law is this: Bezos’ law is the first law that isn’t anchored in technical innovation. Rather, Bezos’ law is anchored in confidence and market dynamics, and will only hold true so long as Amazon is not the aggregate dominant force in Cloud Computing (50%+ market share). Monopolies don’t cut prices.
So where does that leave us? I suppose that depends on how much you love infrastructure. If you love it, this is probably irrelevant to you, but if you don’t this means that the cost of your application will be cut in half every year and a half. That’s a big deal.
Conclusions and/or Considerations
If you’re a programmer, and if you believe in these predictions and the approaches people are taking to improved capacity and performance, your programming style may need to evolve with these changes, too. In short, write Parallel code because it’s better, and realize that the world is migrating towards more cores as quickly as they can because of power consumption patterns and operational considerations.
A Quick Note
At 2600hz, we think a guy named Joe Armstrong nailed it when he wrote the language Erlang. There’s a lot to love, but some of the key highlights for us are trivial parallelization, actor-model with per-actor garbage collection, and massive serialization. We use Erlang as the core of our stack, binding together our Border Controller, Media Servers, Databases, and Applications. It’s not always about the language you use but in our use case, we’ve found Erlang to be scalable, modular, and ultimately fit to the task of managing Global Communications Infrastructures. We feel that this tech, along with the engineers implementing it, give us a significant edge in getting to market and staying online. In addition, it’s a rough world out there and in short order companies that can’t parallelize will be left with massive legacy cost structures.
In a nutshell, Parallel code is where it’s at and at 2600hz we leverage Erlang to help get us to many cores. We’re well on the way and we think this is the direction computing is heading. Parallel Computing ensures Moore’s Laws longevity.
Do you think parallel code is the right answer? Are you worried about your legacy communications system not scaling? Talk to us, we can help. Ring us at firstname.lastname@example.org to chat today!
2600hz Expert Q&A Recap
Whew, the first part of this year has really gone quickly. We want to take some time out to reflect on the stuff we’ve discussed over the last few weeks and to talk about where the Expert Q&A is going from here.
The first Q&A session we did was on Virtualization. This is a huge topic, but we focused on Virtual Machine Timing because that has such a big impact on communications applications. Lots of great stuff in this Q&A featuring our friends from Voxeo and PSSC Labs.
The second Q&A session we did was on Faxing. Again, another huge topic but we decided to go after Faxing because it impacts almost all of our clients. What was really interesting to me about this was the idea that the delimiter in faxing is silence, as opposed to most IP protocols where signaling comes from packets. It’s very clever and I encourage you to dive into this presentation, it’s Jam Packed!
The third Q&A session we did was on Provisioning. We think the industry can do better when it comes to provisioning. There isn’t one provisioner that handles all handsets, but there should be, and that’s what we’re working on at 2600hz. This Q&A presentation explains why this is such a hard thing to do and how our team is solving this problem. This presentation features Andrew Nagy of Provisioner.net fame!
The third Q&A session we did was on Database. Could we have picked a more massive topic? Maybe, but we decided to focus critically on Layer 1 Failures and Network Partitions, which are some of the biggest issues operationally for distributed systems. Featuring Sam Bisbee from Cloudant, we dive headfirst into a TON of content on Databases. This one is not to be missed, it’s a great deck and a ton of awesome commentary. (The image at the top of this email is about Zombie, Flapping Datacenters, which have to be removed from the cluster).
Where to now?
First up we’ve got two more Awesome Q&A sessions scheduled. The one Next Friday, the 14th of June is on Session Border Controllers! If you can’t tell the difference between an SBC and a Firewall, this is the presentation for you.
Two weeks after that we’re covering DTMF! Ever wonder where those tones on your phone came from? We’re going to give you the answers.
If you want to resell 2600hz, we have two great trainings coming up. The first one is this Friday, the 7th of June and the second one is two weeks later on Friday, the 21st of June.
Last but not least, our sales team just got a new shiny 800 #. Ring them at:
Thanks, and we look forward to bringing you tons of new content. Be on the lookout for an announcement of our API Trainings which should be starting later this month or early next!
How Squirrels break Datacenters and other Database Conjectures
This Q&A presentation was influenced by Kyle Kingsbury’s work on Jepsen, an exploration of modern databases. If you haven’t seen his work and you like this stuff, you should go check it out. It’s awesome.
We just did our Epic Database Expert Q&A featuring Sam Bisbee of Cloudant and Darren Schreiber of 2600hz. We covered a range of topics but focused on these three kinds of failures:
- Network Partitions
- Layer 1 Disasters
- Flapping Internet (Special Class of Network Partitions)
All public networks are unreliable; such is the reality of modern distributed database management (and let’s face it, because of AWS we’re all managing distributed databases, whether we like it or not). Sometimes, when these unreliable networks break down, a partition can form. These partitions, depending on your database configuration, can wreak havoc across a wide gamut of scenarios.
Arguably, most of what a database admin does is prepare for network partitions and how to resolve them.
-Joshua Goldbard, 2600hz
Yes, modern databases run fairly well when they’re not in a failure state, but, frankly, the only thing that matters is the failure state. During a partition, it’s important to understand your database behavior, which can vary wildly. At 2600hz, we leverage BigCouch which is a Master-Master replication strategy with Dynamo Quorum(PPT LINK). What that means in plain English is that every node is a master node and it uses consistent hashing to redistribute the load in the event of a partition.
The best advice we can give here is to know the failure modes and behavior of your database and understand the partition realities of the software.
-Darren Schreiber, 2600hz
Layer 1 Disasters
Hurricanes, Earthquakes or Squirrels? Squirrels eating glass. Squirrels caught in HVAC units. Squirrels tampering with Power lines. All of these are examples of Layer 1 Disasters, but we only think about the really massive outages, not the unexpected ones that effect critical infrastructure.
Darren, the 2600hz CEO, has a lot of experience managing Datacenters. Here’s a quick story from back-in-the-day about managing racks in a DC:
Once upon a time a Datacenter vendor decided to give my company a couple of months notice that they were going to 10x our rates. They assumed we couldn’t migrate out of that Datacenter easily, and they were right. Because we were cheap, we did everything ourselves, which meant loading the racks into a pickup truck by hand that we drove in the rain to another Datacenter. Not my definition of Fun.
-Darren Schreiber, 2600hz
Contrast that with our experience during Sandy, when we were using BigCouch:
On the day before the storm, we just turned off the Datacenter. That was it.
We can evade storms, earthquakes and Squirrels because of Cloudant.
-Darren Schreiber, 2600hz
If a Datacenter gets into a Layer 1 issue, we just kill it and move on. When the disaster is mitigated we bring the service back up, but losing an entire DC (or even multiiple DCs) is not an issue because of our database choice.
Protip: If you can’t predict disasters, have a plan to avoid them.
It is up or is it down? Flapping internet is a special case of the Network Partition. Basically, a flapping connection is one that goes down, then comes up, then goes down; this is actually worse than a server going hard down because of the reconciliation process that happens when the networks reconnect. We’ve got one answer for this and one only: Zombie Servers get Double Tapped.
Basically, if a DataCenter is flapping, it’s better to just disconnect that datacenter manually until it can be confirmed as restored. There’s no easy way to say this, Flapping is one of those scenarios that requires manual intervention. If the DC is flapping you have to take it out or you may never get back online.
Protip: If it flaps, Double Tap.
Darren chose to use the last few minutes to pontificate about how ridiculous life was before BigCouch. There was a point very early on where we simply could not get BigCouch to work and we thought we might have to fold the company. Thanks to incredible support from the Cloudant team, we got everything working and the rest is history.
It’s night and day. We just don’t spend any time on the database anymore… We just don’t have problems with the Software.
-Darren Schreiber, 2600hz
Sam chose to talk about right reliability, specifically in the way in which other systems buffer writes and respond to concerns about right availability.
There are a lot of other databases out there that will reply “Write confirmed” when you buffer the write, NOT when it actually commits to disk. The practical effect of this is that if the disk dies before the write moves out of the buffer, you’re missing writes, which is death to a database.
Durable databases write to disk and confirm, they don’t just buffer. Databases that buffer can be very dangerous depending on the workload.
-Sam Bisbee, Cloudant
We had a blast doing this presentation with Cloudant and we can’t wait for the next Q&A Session on Border Controllers in two weeks. Click here to join us!
Two weeks after that, we’re going to discuss DTMF and how all of that nonsense works in VoIP. Register for free here:
Lastly, if you’d like to talk to our friends at Cloudant, check them out at Cloudant.com or in IRC on Freenode in channel #Cloudant.
Thanks so much for checking out our Q&A. If this all sounds like it’s too much work, you should call our Sales team at 8554642600 or email@example.com. We power some of the biggest infrastructures on the planet and we’d love to talk about how we can help your business eliminate the pains of operating communications infrastructure :).
When Darren isn’t busy working on stuff in the guts of the world’s biggest Telecom Infrastructures, he’s helping to write books about FreeSWITCH with the epic FreeSWITCH team. Their latest work is available now!
You can buy the book here: http://www.packtpub.com/freeswitch-1-2/book
Learn more about FreeSWITCH by checking out their site: http://freeswitch.org
Thanks for letting us be a part of such an awesome open-source project!! This is the culmination of a lot of very hard work from our friends Michael, Raymond and Anthony, without whom this project would be impossible.
The much requested deck by James Aimonetti from Kamailio World earlier this year (Epic Powerpoint)
Every now and again we let our engineers out of the office to wreak havoc on the conference circuit. James gave an awesome presentation at Kamailio World earlier this year and we’ve had a bunch of requests for the deck.
SO, without further adieu, I’d like to present “Kazoo” by James Aimonetti, presented earlier this year at Kamailio World (THANKS FOR THE INVITE DANIEL!!!)
2600hz Kazoo Kamailio Integration Deck from Kamailio World by James Aimonetti
Do you like what you’re reading here? Is this the kind of problem you’re trying to tackle? Send us an email at firstname.lastname@example.org to talk about how we can help you scale, or if you’d like to solve these problems, send us an email at email@example.com. We’re looking for engineers, hackers, and people that are really passionate about communications. What are you waiting for, send us an email!!
Provisioning Sucks (And here’s what 2600hz is doing about it)
This is the recap of our Provisioning Q&A session featuring Andrew Nagy of Provisioner.net and Francis Genet, lead engineer on the 2600hz Provisioning project. If you dig this kind of stuff you should check out our next Q&A on Database Management here.
If you have ever dealt with phones, chances are you hate provisioning. If you do it manually, it is exceedingly tedious. If you do it automatically, it can be disastrous. Many organizations opt for a homogenous equipment mix, supporting only one Manufacturer with a proprietary provisioning solution because it works and that’s good enough.
Here at 2600hz, almost all of our clients run heterogeneous infrastructures, which means we have to handle all different manufacturers so we couldn’t use the proprietary solution. Second, we work with a lot of handsets and we realized pretty early on that manual provisioning wouldn’t work for us. So we did what any self-respecting group of telecom engineers would do: we built our own provisioner! And, since we’re awesome open-source citizens, we’ve made the code publicly available HERE too! Let’s take a look at the work we’ve done and why we’re doing it:
On the shoulders of Giants
It’s worth mentioning that we are hardly the only organization to wrestle with the realities of provisioning. Our work is based on Andrew Nagy’s Provisioner.net and, to quote Isaac Newton or Linus Torvalds, depending on who you ask, “If we have seen further, it is because we stand on the shoulders of Giants”. Before we start diving into what we consider the state-of-the-art, we’d like to acknowledge the great work our predecessors have done in bringing us to the point where what we’d like to achieve is possible. Alright, let’s dive in!
Why is this hard?
(Quick note: Cisco Handsets take up to 2.1 hours per phone to provision. That’s why having auto-provisioning is so important. Source)
It’s actually not that hard to provision a single phone, or even 100 phones. Hell, it’s not that hard to provision 1000 phones if they’re at the same site and the same manufacturer. See, routers have this awesome option called DHCP Option66 which lets you point phones en masse towards a provisioning server. All of the devices that connect to the router will receive a URL in a packet header that points the phone towards the config files. This is how the process works, but it’s worth diving into how this works over the WAN in a little more detail. Let’s lay out the process for setting up a handset over a Wide Area Network:
- Phone arrives brand new from factory
- Phone has Provisioning URL added to the on-Device GUI <—- This is DHCP66
- Provisioning server creates a provisioning profile for the handset containing all of the configuration files (MAC Address used for identification)
- The Phone is attached to the corporate network and attempts to connect to the provisioning URL in the GUI
- The provisioning server recognizes the MAC ID of the handset and sends the corresponding configuration files after authenticating the phone
- The phone receives the firmware and if this is a secure environment, performs a checksum on the configuration files to make sure they match
- If everything is Kosher, the phone will begin the update process. Once complete it will enter service.
- Every few minutes (days) or when the phone powers on, it will repeat this process starting at step 4
This process has to work every time for every handset. Now, one would think that after the 150 years of telecom that we’ve had, there might be some standardization between vendors but that’s certainly not the case with respect to provisioning. Every manufacturer has a different way crafting their provisioning files, even down to the number required to boot a phone or even the names of the files. It’s enough to drive a developer batty, but this is what we have to work with in Telecom. Seriously go look at the Polycom firmware grid; it’s like a forest of incompatible firmwares.
The Polycom Nightmare Grid
If you want to present your users with simple-to-consume services, you must first conceal complexity. That’s a recurring theme in all the work we do at 2600hz, but it’s perhaps no more true than what we’re doing with respect to provisioning.
What are we doing about it?
At 2600hz, we believe in presenting simple interfaces for complex services. When we think of provisioning, we want our clients to experience a service that “just works”. We don’t allow folks to see firmware file names because we know what works with our servers. Power users can get this functionality back with trivial difficulty, but for the majority of use cases, the default settings are perfect. Here’s what our provisioning interface looks like in our GUI:
You’ll notice that we request a user to select a make and model of their phone, a name and a MAC address. The only piece of really specialized information is the MAC address; everything else is immediately obvious to the user. But provisioning the handset doesn’t govern how the handset might interact with the network. That’s why we’ve included some extra tools to take the experience just a bit further. Like take segregating Voice and Data traffic without physically separate ports. That’s hard to do without VLAN tags but who wants to manually go into each phone to program a VLAN? That’s complicated, and remember, 2600hz is all about hiding complexity:
Here you’ll see a place to enter a VLAN tag. It really is that simple to push VLANs to all of your clients equipment.
How do we hide all this complexity?
When you check new boxes in the management interface for provisioning your handset, we make on the fly changes to the provisioning file for that phone. If you want to have a Yealink T-22 change from 1 line to 2 lines, you can execute that change NOT with a site visit to your client, but with the click of a mouse. This dramatically reduces labor and wasted time in client site visits by eliminating unnecessary troubleshooting.
2600hz has built an awesome suite of provisioning tools for our clients to use in managing their systems. Provisioning is hard because hardware manufacturers make it hard, but that’s why there’s an opportunity for us to innovate in the first place. By concealing complexity from our clients, we make things run smoother and in a much more controllable fashion.
See our Powerpoint here:
Do provisioning servers make you feel weak in the knees? Does the prospect of reading SIP Packets for a living intrigue you? You might have a future working with 2600hz. If this is interesting, shoot us an email at firstname.lastname@example.org and we’ll chat :D.
Expert Q&A Faxing Edition
We wrapped up an excellent Q&A this morning on the subject of faxing. Yes, the Darth Vader of VoIP has been slain. We broke down a ton of helpful information in our attached powerpoint and hopefully we were able to debunk a lot of myths.
If you missed the event, here are some great quotes:
Silence is the signaling protocol for Faxing
-Darren Schreiber, CEO, 2600hz
Fax machines were designed to deal with noise, not dropouts. IP is designed to deal with dropouts but not noise. The secret to doing faxing at scale is finding the happy medium.
-Joshua Goldbard, VP of Marketing, 2600hz
We also had some great “Pro Tips” for folks doing faxing at scale:
- Turn up the Jitter Buffer
- Turn off the adaptive buffer (Variable timing is death for faxing)
- Turn off Echo Cancellation (Remember silence is the delimiter)
- Reduce Confounding Factors (If it doesn’t help, eliminate it so it can’t break later)
Finally, one piece of the discussion I wanted to pull out was the talk on codec negotiation. Some carriers will ask folks to start calls as G729 and then change to T.38 midway through the call. The reason a carrier does this is because running calls over expensive gear is expensive. Ideally, a carrier will only run the calls that require expensive gear over the expensive gear and everything else should run over the cheap gear, but the one of the major features of the expensive gear is T.38 tolerance. What this means is that you have to first send a signal to your carrier to get onto the expensive switch (starting the call in G729) and then convert the call to T.38 afterwards.
THE REASON CALLS WITH NAKED T.38 FAIL is because many carriers are not setup to route T.38 media unless you tell them it goes on the expensive gear. Therefore, you have to signal G729 first to get on the expensive gear before you can begin a fax transmission. This is a huge pain, but it exists to lower the costs of carrier operations.
Last note: Asterisk and Freeswitch both use spandsp so if someone tells you it works better on one platform over the other, they’re probably commenting about the network and not the switch
Don’t forget our provisioning Q&A, two fridays from today. Here’s the link:
Thanks and see you soon!