Voice in the mountains

Where are we going from here?
— 7:40 PM on May 11, 2008

How can we make voice communications available to rural communities in mountainous regions of the developing world? I have been searching out answers to this question for almost two years now. For the past several months I have been researching and blogging about some of the technological aspects, focusing specifically on long-distance WiFi to determine the feasibility and cost involved and what resources are required. I have yet to research other technologies which I expect are needed, such as VoIP, bridging with existing telephone systems, wireless mesh networking, power systems, and building towers. In the next couple years I will dig into these aspects and hope to implement my research and experiences.

Options: the improbable and the possible

Following are several technologies that have been considered for developing regions, with some of their drawbacks noted.

  • Obviously the mature and well-known option for voice communication in most of the world is cellular telephone technology. However, cellular coverage is largely limited to urban areas, especially in the developing world. As Intel researchers state it (PDF), "existing infrastructure that is useful in high-density areas, where costs can be spread over many users (notably, cellular telephony), is not appropriate for sparsely populated rural regions." For example, the coverage map for Mali, Africa shows nine small pockets where cell phone service is available. The village of Gono, Mali receives coverage primarily because they happen to be close to an urban center. Nevertheless, this story is an example of how rural areas greatly value communications and are often willing to pay for it.
  • Satellite connectivity offers opportunities for rural areas, and a number of mobile satellite-based solutions exist such as satellite telephone and R-BGAN. However, these incur heavy per-minute charges.
  • Using VoIP over a satellite Internet (VSAT) connection is more realistic, but set up and monthly charges are high, and latency (packet delay) makes the voice quality less than desirable.
  • Laying underground fiber optic cable is prohibitively expensive.
  • HF, VHF, and UHF radios require special licensing and are by their nature one-to-many.

The solution I have decided to pursue is a combination of long-distance WiFi, VoIP, and mesh networking. I am not alone in thinking that economical WiFi solutions exist. Organizations such as Green WiFi and Inveneo have invested in this idea, as well as projects like AirJaldi, Nepal Wireless Networking Project, Africa Mpumulanga Mesh, Freifunk, Wireless Networking in the Developing World, and TIER (UC Berkeley) working together with Intel. While consumer-grade WiFi systems may not have the reliability, bandwidth, and quality of service generally expected by cellular service providers, communities with zero voice communication options have much lower expectations, and there are several significant advantages over traditional cellular:

  • Consumer WiFi equipment is readily available and inexpensive because it is so widely used across the world.
  • WiFi equipment usually has low power requirements, which means towers can be solar-powered and not require generators and/or connection to the power grid. As a consequence, initial capital investment and maintenance can be dramatically lower, and no pre-existing wired infrastructure is necessary.
  • WiFi (2.4 GHz, 5 GHz) and other ISM frequencies which are unlicensed in western countries are often also unlicensed in developing countries, so government licensing fees can be dramatically lower.

Testing and lessons learned

During my time in the U.S., I set out to build a mini prototype network which included three nodes, two of which would be over 100 km (60 miles) apart, a VoIP server, and telephones so that I could test every aspect of the network which I envisioned. My first mistake was being overly optimistic, (something I'm good at in other areas as well); I was able to test only a fraction of the planned prototype network. I had planned on using exclusively Linksys routers because they are familiar to me, inexpensive, and ubiquitous. However, as I began researching software and hardware and making purchases, I discovered that, as Yahel Ben-David (of AirJaldi) expressed it, "[Linksys devices] are not advisable for that task. Too little power and worse - it uses a Broadcom radio for which we can't do many of the things we do for Atheros-based radios." This was my biggest course change, and effected many other aspects of the network I ended up testing. However, I am so thankful that I made this change before going up on the mountain.

Thanks primarily to comments made in this blog, the second most important thing I learned involved dealing with snow and ice when installing antennas on mountains. I had the expectation that these grid-type parabolic antennas would do fine in all weather conditions, but I learned (the easy way) that a radome or shelter of some sort is essential. All my testing was well worth it, although the job was a lot longer and harder than I expected. My biggest disappointment was that the 121 km (75 mile) link only achieved about 585 Kb/s throughput--plenty for a good number of VoIP phone calls, but I was hoping for several times more. Also disappointing was the fact that the 23 km link never worked as planned, and that I could not test VoIP at all over the long-distance link. Since the January testing, Intel made an exciting announcement about a router capable of greater than 6 Mb/s over 100 km. Although it is not mentioned in the article, Intel has been partnering with the UC Berkeley TIER group to design the software for this router--the very software I used in my 121 km test.

Going forward

Technology alone is not sufficient. For any project like this to work for more than a couple of years, it must have a sustainable business model. (In the long run, at least as much money needs to come in as is going out.) Village Phone, which builds on traditional cell phone technology, has been very successful in bringing communications to rural Africa. Their model, in summary, involves an entrepreneur from the village purchasing a cell phone, roof antenna, and charger with the help of a microloan. They are then able to sell minutes to villagers for a profit. The cell phone antenna must be within about 35 km of a cell phone tower and have line of sight, thus making the technological aspect of this model unworkable in many rural or mountainous regions. The business model, however, could potentially be used just as successfully with other technologies, including WiFi paired with VoIP. This area will require much more research--learning about small businesses, understanding local culture, and relating to local and national governments.

In all of this, I would love your input. I really appreciate and have gleaned a lot from the people who have taken the time to write, both in personal emails and in the blog comments. I have worked with computers for over twenty years, but my experience in most of the other areas is quite limited. With your help, this project has a much higher probability of success.

31 comments — Last by sagisys at 12:39 PM on 05/16/08

Retest day
— 9:15 AM on January 29, 2008

January 17, 2008


We were finally able to make it up to the mountains again to finish what we had started, and I'll go ahead and say that the testing was successful. In the photo above of node 14, one of those distant bumps in the bright clouds must be the snow-capped mountain of node 13, 121 km (75 miles) away. You can see that we were able to get the weather to cooperate this time; the sun was nice.

There was plenty of snow on the road to node 14, and we would not have made it half way up the mountain without a 4x4 pickup. The road to node 13 was completely blocked by a snow drift near the top, so the test equipment had to be carried the last quarter mile by hand. Fortunately, the base, steel mast, and antenna had been left at the site. Another problem at node 13 was that the mast had fallen down and one of the cable tensioners was missing. It must have turned and come unscrewed. Fortunately, no damage was done.

After aiming both antennas with the compass and inclinometer again, the signal level at node 13 was -61 dBm. We tweaked both antennas vertically and horizontally while watching the signal level with iwconfig, but were only able to improve it slightly (-59 dBm). We did see -57 dBm once. At node 14, the received signal began at about -69 dBm and got as good as -63 dBm at one point. The level seemed to vary a little even when the antennas were not being adjusted, and weakened by about 4 dBm during the time we had the connection up.

We measured connection speed with iperf running on the actual TIER Linux routers. TCP throughput averaged about 585 Kb/s (one direction). We simulated VoIP traffic with small UDP packets, and achieved about 300 Kb/s in both directions simultaneously. This should support a good number of voice connections. Ermanno et al achieved 3 Mb/s over 279 km using very similar hardware and software (see my prior blog entry), so I know that much faster speeds are possible. This will need further investigation.

After completing the performance tests with the TIER Linux routers, we disconnected them and connected the Linksys routers which had been configured for nodes 10 and 13. No configuration changes were made; we just plugged them in. I was surprised that it actually worked, although throughput was very slow (about 91 Kb/s). We also figured out why the link between nodes 10 and 13 never worked--the wrong antenna connector (on the back of the Linksys box) was configured! I had tested this and was certain that the setting was correct, but the 121 km link only worked with the Linksys devices when the antennas were connected to the opposite connector than the one we had been using.

During the testing, the team at node 13 met a guy who maintains some of the professional equipment mounted on one of the towers there. He mentioned that he has given up on the grid-type antennas for high elevations because ice accumulations warp the parabolic dish and degrade performance. He recommended solid antennas with a cover (radome) to keep ice and snow off the dish, and specifically Andrew antennas, which the snow just slides off of. They cost about twice as much, but last much longer.

I am very glad to have this phase of research and testing completed, and to know that 121 km is feasible. It is a great alternative to VSAT for certain situations.

4 comments — Last by eitje at 10:53 PM on 02/19/08

Test day
— 12:38 PM on January 14, 2008

December 6, 2007

Because of the problem with node 10 from the day before, I removed the node 10 antenna and remounted it on the other side of the pole, thinking that the problem may be the way the coaxial feed cable was hitting the metal pole. I then aimed it again, double checked everything, and made sure it was plugged in.

Early in the morning (not early enough) we loaded one truck and sent three guys off toward node 14, while two of us drove to node 13 in another truck. At the node 13 site, we hooked up the antenna aimed at node 10 again (aka the 1310 antenna) and checked for a signal. Unfortunately, there was still no signal at all, even after re-aiming and re-checking everything. Based on the testing the day before, it seemed that there might still be a problem with the node 10 antenna or cable. I wonder if the way the antenna feed cable had been hitting the metal pole actually made a permanent kink in the cable, causing significant signal degradation. I have read that kinking coaxial antenna cables can easily ruin them. There are certainly other things which could have gone wrong too, such as the antenna aiming, computations for antenna aiming, computations for signal level at that distance, fog, etc. We had to abandon the connection between nodes 10 and 13, which meant that we could not access the Internet, and therefore could not do some of the VoIP testing which had been planned.

On all four antennas, aiming was done with an inclinometer and compass. I realize that in some cases the antenna's strongest signal may not be in exactly the same direction as the feed horn points, but it is a good starting point. For the 30 dBi antennas, we first used an inclinometer to adjust the vertical angle using the bolts included on the antenna mount, and then a compass to adjust the horizontal alignment with bolts which positioned a wooden block at the base of the mast and caused the entire mast to rotate. This method would not work if multiple antennas were mounted on the same steel pipe; is there a better way?

For the 24 dBi antennas, the mount did not include a tilt adjustment, and we did not build a rotatable mast, so both the horizontal and the vertical had to be adjusted simultaneously and then the bolts tightened carefully while hoping that the antenna did not move. There must be a better way to do this.

Because of an abundance of metal, the compass could not be close to the antenna. Two methods were used. For three of the antennas, we tied a string to the mount, ran it through the antenna at a small slit directly above the base of the feed horn (where the two halves of the parabolic dish join), and then out about two meters directly in front of the antenna. (This method wouldn't work if the antenna were mounted on a tower! How do telecom professionals aim antennas?) We then pulled the string tight, held the compass over the end of the string, and moved the string left or right until the string was lined up with the compass and the needle pointed at the computed compass direction. While holding this position, another person rotated the antenna until the feed horn was aligned with the string. For one of the 30 dBi antennas, a very similar process was used except that a mirror on the compass substituted for the string.

If anyone is 'trying this at home', the above technique will hopefully provide a signal strong enough to be detected by the radio card. Once this is achieved, iwconfig or other software can be used to fine tune the alignment. Another method for aiming, which could be used in addition to or perhaps instead of the above-mentioned method, uses a signal generator at one end and a spectrum analyzer at the other.

Before telling about the test results, I want to briefly mention some of the obstacles we faced (besides the technical ones and those already mentioned). First, coordinating schedules between the five of us was difficult because of things like preexisting commitments, constant threats of jury duty, and trying to avoid cold and rain (and we failed miserably at avoiding the cold and rain). Second, the 1310 antenna mast was unstable because we didn't take the time to make a solid cement base as we had done for the two 30 dBi antennas. Third, we miscommunicated about which power plug was needed to run a Toshiba laptop off of a 12 volt battery, and therefore had limited use of the laptop because its internal battery did not work well. Fourth, we had problems installing iperf on the Toshiba laptop and had left this step to the last minute, and thus could not do the connection speed tests which we had planned. Fifth, we tried a new Google Maps route to node 14 which should have been much faster but instead led to a locked gate. The men had to backtrack and take the route we had used before. Finally, as expected, everything took longer than expected, and we ran out of time. Because of this and the rain, we actually were not able to do any speed tests. Most of these problems could have been avoided by testing everything ahead of time with the actual equipment we had planned to use, or at least having backup equipment and plans. But testing takes a lot of time too.

The good news is that, against all odds, the 121 km connection worked! After aiming the two 30 dBi antennas with only the inclinometer and compass (no signal generator, spectrum analyzer, or tweaking with iwconfig), we plugged in both radios at 15:13 local time and established a stable link. iwconfig reported a signal level of -55 dBm at node 13 (-60 dBm at node 14). We did not have time to keep the link up for long, but we did log 424 pings (1 lost, 3.726 ms round trip average) and were able to establish a stable ssh session.

We plan to test again soon and hope to see if we can improve on the signal level by fine tuning the aim of the antennas, as well as perform a number of throughput and jitter tests with iperf to help determine how many phone calls the connection can support.

5 comments — Last by wirelesh at 5:38 PM on 01/18/08

Final test day preparations
— 2:57 PM on January 4, 2008

December 5, 2007

Click on diagram for a larger version

The diagram above details the logical configuration of the network we are setting up (some of the data has been intentionally modified). It includes two custom-built routers described in my last post, optimized for long-distance links, and two Linksys WRT54GL routers running Freifunk firmware 1.6.20. Freifunk is essentially OpenWrt with a modified implementation of OLSR mesh software, some testing and management tools, and a friendly GUI. I wanted to include Freifunk in my testing because a mesh network should work well in the mountainous terrain to provide reliable wireless connections between villages which do not have line of sight, and the WRT54GL is a low-cost, low-power, and flexible platform on which to build routers.

The network diagram was not just done for the benefit of this blog. During the setting up and testing, I found it an invaluable reference source--all of the numbers I needed in one place, and presented visually. It is done in Gimp, allowing easy rearrangement of elements and changes to the text blocks.

Note how the sum of corresponding elevation angles is not zero; this is due to the Earth's curvature. In fact, on the long link, both antennas are pointed slightly down. Similarly, compass directions for corresponding antennas are not 180 degrees apart due to differences in magnetic declination at the endpoints, although the differences are negligible. Radio Mobile computed all of these values for me, except for adjusting for declination.

After configuring all four routers, I connected node 10 to the Internet, connected the actual power and networking cables and antennas (but without the parabolic grid parts), and placed the three nodes a few meters apart. This part of the testing is very important as it verifies the operation of all of the hardware and software components as a whole while you still have access to spare parts, a good work area, electricity, Internet, etc. More information on staging can be found here.

On December 5, two of us drove up to the location for node 13. The plan was to set up the bases, poles, guy wires, and two antennas, and verify the connection between nodes 10 and 13 (named 1013). The equipment planned for node 14 was set up at the node 10 site (with the antenna aimed at node 13), and we had hoped to test the 1314 link over 23 km. The following day we would set up node 14 and test the 121 km link.

After arriving and unloading, the first problem we faced was pounding our fence posts into the ground for anchoring the guy wires. What looked like good soil all over was only a thin layer of dirt on top of fairly solid rock. We were fortunately able to work around this by using three sharpened angle-iron stakes (which we could pound into the rock) and wrapping wire around two existing rocks. The rocks happened to be nicely placed so that we could still avoid having a guy wire in front of either antenna. While trying to carry another large rock over to help brace one of the existing rocks, I injured my leg and wrist, but not so badly as to hinder the work that day.

The second problem was that we could not connect with node 10; there was no detectable signal at all. After talking with someone at node 10 (via cell phone), we realized that the node 10 router had not been plugged in that morning. However, plugging it in still did not allow us to connect. The signal (SSID 1013) was detectable near node 10 using a laptop, but not at our location (using the 24 dBi antenna). We then turned on the node 14 radio located at the node 10 site, which was already aimed at node 13. This signal (SSID 1314) was received at node 13 using the same 24 dBi antenna and Linksys router, and the signal level was reasonably strong. We therefore concluded that the problem was with the node 10 antenna and would have to wait.

I also took the opportunity to check for possible sources of interference with a Wi-Spy 2.4x spectrum analyzer. When connected to the 30 dBi antenna pointed at the site for node 14, there was a very strong and constant signal around channel 13 (which requires a license in the U.S.), but the other channels were reasonably quiet, especially around channel 1. When connected to the 24 dBi antenna pointed at node 10, it was also reasonably quiet. When connected to a short omni antenna, it was very quiet, so I was not concerned about interference from or interfering with existing equipment.

For anyone interested, here is my packing list (I like lists). Construction materials: routers; antennas (dish, feed horn reflector, feed horn, mounting hardware, instructions); pigtales; car battery (fully charged); power cables from battery; mast, base, guy wires, turnbuckles, cable clamps, guy wire anchors; cable ties (attach Soekris box to pole, secure cables, keep anchor cables from slipping); rain cover; Ethernet cables.

Tools for the Hyperlink 24 dBi dish: two 8 mm tools, 10 mm tool, 10 mm deep well socket; small phillips screwdriver.

Tools for the Hyperlink 30 dBi dish: two 17 mm tools, 14 mm deep well or wrench, 12 mm tool, two 25 mm or adjustable wrenches; small phillips screwdriver.

Tools for the mast: hammer or sledge hammer (stakes into ground); large pliers (stakes out of ground); small pliers; screw driver (tighten turnbuckle cable tensioners); shovel; work gloves; wire cutters.

Other tools: bin or tub for small parts while assembling; 1/4-inch and 3/8-inch nut drivers; volt meter; duct tape; driving directions; cell phone, numbers of all other parties; food, water (it will be a long day); camera, spare batteries; binoculars; GPS; compass; inclinometer (make pole vertical, aim antenna at proper angle); azimuth and compass bearing of neighbor nodes; string for aiming; laptop (Radio Mobile, Kismet, spectrum analyzer); laptop car adapter, cables; business cards (if asked about what we're doing); permission letter (node 13); gate key (node 13); boots, old jeans, jacket; spare router; extra steel cable, cable hardware, baling wire; rope (you never know); spare AA batteries; umbrella (shade laptop screen from rain or sun); spectrum analyzer; blankets for transporting antennas; pen and notebook.

6 comments — Last by radix at 12:50 PM on 01/11/08

Setting up long distance routers
— 7:53 PM on December 13, 2007

November 24, 2007

I had originally planned on using inexpensive Linksys WRT54GL devices for the long distance link. This has been done successfully in the past on much longer links (see my previous post). However, I did not realize how seriously throughput is effected by distance using the unmodified 802.11 protocol. On a 279 km link with Linksys hardware, they were only able to get about 70 Kb/s, which will not support too many VoIP calls! As Yahel Ben-David (of AirJaldi) expressed it, "I would not call 70kbs 'bandwidth'." In an exchange of emails, Yahel convinced me to use the software developed by the TIER group instead of WRT54GL devices--"WRT54GL are not advisable for that task. Too little power and worse - it uses a Broadcom radio for which we can't do many of the things we do for Atheros-based radios." He also pointed out what I have come to know all too well--"The fact that [there] is little technical documentation about the WiLD project remains, and IMHO is a big problem. Nevertheless, things are not so-ready for production quality usage, again IMHO - although lots of work is being done and should soon be ready for wider consumption."  (WiLD stands for wireless long distance.)

Based on the decision to use TIER Linux, I purchased two Soekris net4826 single board computers, two Ubiquiti XR2 mini-PCI radios, two aluminum outdoor cases, and several assorted pigtales and adapters.

Flashing the Soekris boards with TIER Linux proved very tedious for me. The TIER site has good directions for writing TIER Linux to boards with removable flash memory, such as the (now discontinued) WRAP boards. However, the net4826 has flash memory which is soldered to the board and cannot be removed for programming. Because there are no connectors for external drives, the only way to flash the memory is to boot via PXE (over the network). I waded my way through connecting via the serial port (using this USB adapter and null modem cable), setting up DHCP and TFTP servers, finding a suitable PXE Linux loader image, modifying the TIER Linux install script to create the image on a loopback device with the proper drive geometry (//sfdisk// and lilo don't work the same with loopback devices, and losetup -o doesn't have an option for specifying the end of a partition), and flashing the image onto the Soekris board via wget and dd. Most of these steps took several attempts, but now I have clear documentation on how to do it all again, and I learned a lot in the process.

The above was using TIER Linux 2.4.26, which does not seem to have options for tuning parameters such as the number of retransmissions and forward error correction (FEC) discussed in this paper. (If these options are available, they are not documented.) FEC may be especially useful for reducing jitter, which is one of the key requirements for high-quality VoIP calls. I tried version, which may include these options, but for some reason it did not recognize the XR2 radio card. When time permits, I'll investigate this option further.

Installing the boards into the above-mentioned cases was much easier, using #4-40 15 mm screws and 7 mm plastic spacers. I punched out holes for the N connector and for a CAT5 cable. Perhaps not for this test, but in the future I'd like to try this CAT5 pass thru gland. I realize it's probably less convenient than an actual RJ45 jack, but it's so much cheaper. Has anyone had experience with something like this?

My plan was to run a single CAT5 wire to each Soekris board to supply power as well as Ethernet connectivity. However, I was not able to get either of my Soekris boards to turn on using PoE at 12.8 volts DC. PoE did work with 25.5 volts (using pins 4, 5 [blue] for positive and pins 7, 8 [brown] for negative), but in the end I used 12.8 volts through the DC power jack. Others (here too) have also experienced problems with PoE on these boards.

9 comments — Last by mikeltm at 3:15 PM on 01/03/08

A few long distance links
— 12:08 AM on December 8, 2007

November 8, 2007

I realize I'm not going for any world records with a 100 km WiFi link. Many have gone before me; here is a partial list.

In July, 2005, a group set a then-world-record with a 201 km link. They achieved 11 Mb/s (perhaps link speed) over this distance using huge home-made dishes (3.0 and 3.7 meters) and 300 mW Z-Com PCMCIA radios. More details can be found here, here, and here.

Also in 2005, another group successfully connected a 420 km link between a ground station and a stratospheric balloon. According to Pietrosemoli, they used 6 watt amplifiers to boost the signal strength.

In April, 2006, Ermanno Pietrosemoli and his team used Linksys hardware on a 279 km link (this version has additional photos). They used a signal generator and spectrum analyzer for aiming only, and adjusted the ACK timing using OpenWrt. Based on a screen shot of their file transfers, it looks like they were getting around 70 Kb/s throughput.

In February, 2007, Ermanno et al tested a 101 km link. Here are some pictures of the test. They used 28 dBi solid dish antennas, Soekris boards (and one Metrix board), and software developed by the Berkeley TIER group.

In April, 2007, Ermanno et al achieved another 279 km link, but this time with about 3 Mb/s throughput in each direction. They used commercial HyperLink 30 dBi antennas and software from the TIER group again.

Also in April, 2007, the same team set up a 382 km link using the same hardware. As far as I know, this is the current record without using amplifiers. However, in Ermanno's words, "This link proved to lack stability, and we would loose connection from time to time." More details can be found here and here.

Wikipedia has a short article on this topic.

3 comments — Last by wirelesh at 11:36 PM on 12/12/07

Permission for node 14
— 12:42 AM on December 2, 2007

October 29, 2007

Some effort was required to get permission to do our testing on the node 14 site.  It turns out that the east side of the mountain (the road and the wooded area east of it) is owned by the government, and west of the road is owned by a forest products company.  We were able to get a hold of a representative for the east side and explained what we were planning to do and the long-term project goals.  He said the area is public land and that we were free to drive to the gate (which he thought would be locked, although it turned out to not be), walk up to the top (less than half a mile) and have a look around.  To get permission to do our connectivity testing, however, he would need details about exactly what we would be doing and he would need to talk with others to make sure it would not interfere with the equipment which was operating there.  There are five sites operating on the top of the mountain, including a wireless ISP, so he wanted to be certain that our testing would not interfere.

On our site survey, I concluded that the top of the road, and even a ways down the road where our car was parked, probably have sufficient line of sight to the node 13 site, but the company-owned land west of the road was better suited because it was farther from the existing equipment and had better line of sight.

After a little phone tag, this morning I was able to talk with a representative of the forestry company about what we wanted to do.  She asked a number of questions about what "connectivity testing" was, what we wanted to do, and the organization we were working with.  I explained that the frequencies we were going to use were 'public domain' and should not cause any interference with the equipment there.  She finally reluctantly agree that we could do our testing under the terms they have to allow hunters to use the property.

2 comments — Last by eitje at 8:16 AM on 12/04/07