Napster was built as hybrid centralized/decentralized system for music file sharing, while also providing instant-messaging and group chat for users in its network. Later, the protocol was reverse-engineered and extended by various open-source projects to allow for the transfer of other file types (as well as more advanced technical features). It’s widely agreed upon, in both the technology and the music industries, that Napster was responsible for the collapse of the then-dominant major record labels and their revenue model. It’s relevant to keep this social context in mind, in order to gain a better perspective on the strengths and weaknesses of the Napster (later OpenNap) architecture, and to see it as the first major spread of a new type of social culture-sharing protocol.
Napster was originally developed by an American programmer named Shawn Fanning (left). Sean Parker (right) joined Fanning as a co-founder early on in Napsters development, and the pair launched Napster in June 1999. For reference, Parker later become the President of Facebook at a very early stage, after which the company raised its first large investment. The original Napster.com site was shut down in 2001, after a decisive court ruling. A paid version was then launched, which was later purchased by Rhapsody in 2011.
Napsters architecture is focused on the centralized meta-data server, which coordinates a potentially large number of peers, and a decentralized data/file transfer network, where users transfer mp3 files directly between each other. This approach balanced simplicity of architecture (due to centralization) with increased scalability and reduced cost (with decentralized file transfers).
- Simplicity of architecture comes from the centralized coordination/processing and storage of peers metadata (searches,coordination of p2p flows,chat,etc.). This is achieved with a classical Client/Server model.
- Increased scalability and reduced cost come from the fact that the actual file transfer happens directly between the peers that are connected to the central coordination servers. A significant increase in the number of those peers does not require a large increase in processing power of the server, nor the drastic increase in the consumed server bandwidth - peers are the ones that are donating their bandwidth to share files with other users.
The original proprietary Napster client application was a Windows codebase, developed using C++ and the MFC (Microsoft Foundation Class) API for its user interface. The system was divided into a server part, and client part.
“The backend software that ran the Napster servers and the Napster client were two distinct code bases.
Backend software - the backend consisted of two primary systems: a custom load balancer called controld and the main Napster server called napd. controld was written in C. napd was written in C++. However, embarrassingly, it’s a stretch to call it C++ because it was just C with classes. We didn’t really use any OO concepts. We did end up using some STL, but it played a minor role in the overall system.
Napster client - a Windows MFC application written in C++. Shawn Fanning wrote 99% of the code in the application.”
Ali Aydar - Quora
Ali Aydar - Senior Director of Technology, Napster (1999-2002)
“He is best known as an early employee and key technical contributor at the original Napster.”
“He joined in September 1999, becoming its first non-founding employee.”
“Aydar authored Napster’s search engine software, which supported the millions of search queries Napster users made every day.”
Very early on in the spread/growth of the proprietary Napster system, it was becoming clear to the technical community that this protocol and application had great potential. As a result an OpenNap open-source implementation of the Napster server appeared online, first released in 1999. It was licensed under the GPL license.
It’s important to note that the proprietary Napster protocol was reverse-engineered; there was no official release of its specification by the Napster company. This reverse-engineering work was done by programmer with an online nickname Drscholl, and several of his peers.
The OpenNap server can be compiled for both Windows and UNIX targets. It’s original sourceforge repository no longer exists, but last time it did it represented the version 0.44-BETA (released September 30, 2001). The specs however are still online - opennap sourceforge spec. The WinMX client website also maintains a good OpenNap protocol spec page, with some decent protocol flow diagrams to help visualize things.
On Github OpenNap implementation code is present in 2 repositories, both implementations are in C:
- Scassiba opennap github repo
- Khaytsus opennap github repo - opennap v0.50 (BETA) (released August 29, 2004)
this OpenNap server was not compatible with Napster.com, due to subtle incompatibilities.
Going beyond the mirroring of the protocol implementation of the closed-source official Napster server, OpenNap implements some interesting improvements, namely: sharing of any media type, and the ability to link servers together.
The linking of servers together allows for the creation of OpenNap networks, within which users are be able to use the chat and search functionality across servers. These servers do not have to be all operated by the same entity, or to exist in the same physical network.
With this interlinking of servers we encounter a classic distributed systems problem of maintaining a consistent database across the entire network. In OpenNap this problem is tackled by the propagation of client messages, as they come in from the edge of the network. As client messages are received by some OpenNap server, they are passed to that servers peers, as soon as they come in, to propagate through the network. Server-to-server communication can optionally also be compressed (via zlib) as well.
One of the more powerful features that linked-servers configuration enables is remote browsing (
"remote_browse" option in the config file), where users from one OpenNap server are able to browser directories of (music) files of users that are connected to another OpenNap server. A side-effect of this feature is that it can account for a noticeable increase in the amount of cross-server communication. However even with that penalty this newfound ability to network servers together into larger networks made all the difference for the growing diversity of operators and groups that started hosting OpenNap nodes.
Another feature is that OpenNap servers can be compiled to run in router-only mode. These servers dont allow file-sharing commands for their direct clients, and instead only route messages to servers that are linked to them. To compile the OpenNap server in this router-only mode the
"--enable-router" flag is used for the “configure” stage in the build process, before the actual build of the server binary. There is also the
"auto_link" configuration option, which when set to “on” will force the OpenNap server to attempt linking with all servers that are listed as DNS names in its “servers” file, immediately upon startup.
Around 2001 an indexing service for all OpenNap servers was created, and called Napigator. This allowed for automatic discovery and connecting to listed servers by compatible OpenNap clients. Admins were greatly incentivized to list their servers in this index.
Just a few of the many popular clients for OpenNap were:
- WinMX - OpenNap client that was developed by Frontcode Technologies, which also has its own p2p network separate of OpenNap - wikipedia, WinMX World
- Utatane - Windows OpenNap client developed in Japan - wikipedia
Direct clones of Napster are already everywhere on the Net – many of them overseas and thus beyond the quick reach of U.S. courts. Their numbers are expected to grow as Napster sympathizers work on their own to keep the flame burning.
The Wall Street Journal - 2000 - Internet Is Full of Napster Copycats, And Users Are Eagerly Tuning In
“Things are really crazy. People are looking for an alternative. We are getting swamped with many more downloads than we are used to,” said Chad Boyda, of Dublin, Calif., who wrote a program called Napigator that allows Napsters users to log on to music-sharing computers besides those run by Napster itself.
The Wall Street Journal - 2000 - Internet Is Full of Napster Copycats, And Users Are Eagerly Tuning In
Napster Protocol Messages
As mentioned previously, this spec of the OpenNap protocol was published by drscholl in April 2000, by analyzing network traffic (protocol) of the linux Napster client. It may not be identical to the official Windows client protocol implementation, but for the most part it was compatible.
OpenNap Servers run on ports 8888 and 7777, and there are numerous command-line (CLI) options to configure various parts of their function.
All messages sent by clients and servers are a byte sequence in the form
<type> are both 2-bytes long,
<length> specifies the length of the
<data> part of the message. The
<data> message portion is encoded as a plain ASCII string, and it is the part that varies the most from one message type to another. Also, in many cases strings are encoded with doubel-quotes
"some_song_name.mp3"). Fields in the data portion of the message are also separated by a whitespace
In the bellow listing, I roughly grouped messages by the area of functionality they belong to, for easier review. There are also messages that belong in multiple groups, in particular messages that deal with user data and its interaction with file transfer and chat (instant messaging).
msg_code | msg_type #0 error message #2 login - CLIENT #3 login ack - SERVER #4 version check - CLIENT #5 "auto-upgrade" - SERVER #6 new user login - CLIENT #7 nick check - CLIENT #8 nickname not registered - SERVER #9 nickname already registered - SERVER #10 invalid nickname - SERVER #14 login options - CLIENT #100 client notification of shared file - CLIENT #102 remove file - CLIENT #110 unshare all files - CLIENT #200 client search request - CLIENT #201 search response - SERVER #202 end of search response from server - SERVER #203 download request - CLIENT #204 download ack - SERVER #205 private message to/from another user - CLIENT, SERVER #206 get error - SERVER #207 add hotlist entry - CLIENT #208 hotlist - CLIENT #209 user signon - SERVER #210 user signoff - SERVER #211 browse a user's files - CLIENT #212 browse response - SERVER #213 end of browse list - SERVER #214 server stats - CLIENT, SERVER #215 request resume - CLIENT #216 resume search response - SERVER #217 end of resume search list - SERVER #218 downloading file - CLIENT #219 download complete - CLIENT #220 uploading file - CLIENT #221 upload complete - CLIENT #300 optional ports - CLIENT #301 hotlist ack - SERVER #302 hotlist error - SERVER #303 remove user from hotlist - CLIENT #320 user ignore list - CLIENT, SERVER #321 user ignore list entry - SERVER #322 add user to ignore list - CLIENT, SERVER #323 remove user from ignore list - CLIENT #324 user is not ignored - SERVER #325 user is already ignored - SERVER #326 clear ignore list - CLIENT, SERVER #330 blocked ip list - CLIENT #332 block ip - CLIENT #333 unblock ip - CLIENT #400 join channel - CLIENT #401 part channel - CLIENT, SERVER #402 send public message - CLIENT #403 public message - SERVER #404 error message - SERVER #405 join acknowledge - SERVER #406 join message - SERVER #407 user parted channel - SERVER #408 channel user list entry - SERVER #409 end of channel user list - SERVER #410 channel topic - CLIENT,SERVER #420 channel ban list - CLIENT,SERVER #421 channel ban list entry - SERVER #422 channel ban - CLIENT #423 channel unban - CLIENT #424 channel ban clear - CLIENT #430 invite a user - CLIENT,SERVER #431 invitation accepted - CLIENT #432 invitation declined - CLIENT #500 alternate download request - CLIENT #501 alternate download ack - SERVER #600 request user's link speed - CLIENT #601 link speed response - SERVER #603 whois request - CLIENT #604 whois response - SERVER #605 whowas response - SERVER #606 change user level - CLIENT #607 upload request - SERVER #608 accept upload request - CLIENT #609 accept failed - SERVER #610 kill (disconnect) a user - CLIENT #611 nuke a user - CLIENT #612 ban user - CLIENT #613 set data port for user - CLIENT,SERVER #614 unban user - CLIENT #615 show bans for server - CLIENT #616 (ip?) ban list entry - SERVER #617 list channels - CLIENT,SERVER #618 channel list entry - SERVER #619 queue limit - CLIENT #620 queue limit - SERVER #621 message of the day - CLIENT,SERVER #622 muzzle a user - CLIENT #623 unmuzzle a user - CLIENT #624 un-nuke a user - CLIENT #625 change a user's linespeed - CLIENT #626 data port error - CLIENT,SERVER #627 operator message - CLIENT,SERVER #628 global message - CLIENT,SERVER #629 banned users - SERVER #640 direct browse request - CLIENT,SERVER #641 direct browse accept - CLIENT,SERVER #642 direct browse error - SERVER #652 cloak user - CLIENT #700 change link speed - CLIENT #701 change user password - CLIENT #702 change email address - CLIENT #703 change data port - CLIENT #748 login attempt - SERVER #750 server ping - CLIENT,SERVER #751 ping user - CLIENT,SERVER #752 pong response - CLIENT,SERVER #753 change password for another user - CLIENT #800 reload config - CLIENT #801 server version - CLIENT #810 set config - CLIENT #820 clear channel - CLIENT #821 redirect client to another server - CLIENT,SERVER #822 cycle client - CLIENT,SERVER #823 set channel level - CLIENT #824 emote - CLIENT,SERVER #825 user list entry - SERVER #826 channel limit - CLIENT #827 show all channels - CLIENT,SERVER #828 channel list - SERVER #829 kick user from channel - CLIENT #830 list users in channel - CLIENT,SERVER #831 global user list - CLIENT #870 add files by directory - CLIENT #900 connection test - SERVER #901 listen test - SERVERlegenduser related messagesfile related messageschat related messagesinteresting messagesmsg sent by CLIENTmsg sent by SERVER
OpenNap 0.5 only messages: #612 ban user/ip - CLIENT #10000 client quit - SERVER #10010 server login - SERVER #10011 server login ack - SERVER #10012 request server link list - CLIENT #10013 user ip - SERVER (DEPRECATED) #10014 registration info - SERVER #10018 encapsulated message - SERVER #10019 server link info - SERVER #10020 server quit - SERVER #10021 notify mods - SERVER #10022 server to server pong - SERVER #10023 time check - SERVER #10024 remote whois notification - SERVER #10100 server connect - CLIENT #10101 server disconnect - CLIENT #10110 kill server - CLIENT #10111 remove server - CLIENT (DEPRECATED) #10112 show server links - CLIENT, SERVER #10115 show server stats - CLIENT, SERVER #10116 server ping - DEPRECATED #10117 rehash (reload) configuration files - CLIENT #10118 display client information statistics - CLIENT,SERVER #10119 display which server a user is on - CLIENT #10120 ping all peer servers - CLIENT #10121 who was - CLIENT #10122 mass kill by ip - CLIENT #10123 server command histogram - CLIENT,SERVER #10124 end server histogram - SERVER #10203 set user mode - CLIENT #10204 set channel op - CLIENT #10205 remove channel op - CLIENT #10206 channel op list - CLIENT #10207 drop channel - CLIENT #10208 send message to channel ops - CLIENT #10209 change channel mode - CLIENT,SERVER #10210 invite user to a channel - CLIENT,SERVER #10211 give voice to speak in moderated channel - CLIENT #10212 remove voice to speak in moderated channel - CLIENT #10213 muzzle a user in a specific channel - CLIENT #10214 unmuzzle a user in specific channel - CLIENT #10300 share generic media file - CLIENT #10301 new browse - CLIENT (DEPRECATED) #10302 new browse result - CLIENT (DEPRECATED)
Here is some more detail on a few common message types from the above listing:
[FILENAME CONTAINS “artist name”] MAX_RESULTS “max” [FILENAME CONTAINS “song”] linespeed bitrate freq wma-file local_only
“filename” md5 size bitrate frequency length nick ip link-type [weight]example response:
“random band - random song.mp3” 7d733c1e7419674744768db71bff8bcd 2558199 128 44100 159 lefty 3437166285 4
nickargument), by sending this message to the SERVER.
nick “filename”example request:
lefty 4877911892 6699 “random band - random song.mp3” 10fe9e623b1962da85eea61df7ac1f69 3
#751message. the response value is the name of the user that has been pinged.
Downloading File Flow
As mentioned previously, file transfer happens between clients directly. The servers role is to synchronize the initial exchange of information needed to determine which peer a file needs to be downloaded from, what their IP/port is, and if they’re free to make such a transfer.
To start this download, the initiating client sends a
download request message
#203 to the Server. If the server validates the message and the desired peer can serve a download of that file, the Server will respond with a
download acknowledge message
In the one case the desired peer from which we want to download a file is not firewalled, and TCP connection is established with an IP/port that was specified in the
#204 message. That remote peer (with the desired file) when the connection is established responds with an ASCII character
1 (ASCII 49). After that a string with value
GET is sent to that same target peer. Finaly the p2p file-transfer request message is sent
<length><type><nickname> "<filename>" <offset>.
Once the file transfer from the target peer is started, the initiating peer will send a notification to the server, that its downloading a file, via the
download started notification. When the download is complete the initiating peer sends a
file transfer complete to the server, to indicate that it completed downloading.
Firewalled Downloading is more complex, due to the fact that peers can’t reach each other directly. Instead the OpenNap has to act as intermediary in establishing the connection.
Alternate/Firewall Download Request (link) and
Alternate/Firewalled Donwload Acknowledged Notification (link) messages are used for this flow.
Bellow is an illustration of the stages of downloading a file from one peer to another:
Napster was the first trully popular p2p file-sharing platform. It contained some important decentralized concepts, namely its p2p file transfer function. However, its downfall was rooted in its metadata/coordination centralization, via its Napster servers. This made it easy for unfriendly actors to stop and control the platform. The centralized storage of metadata made searches very efficient, and allowed for keyword based queries (not that its impossible to achieve that in decentralized model). However, Napster lacked immunity against attacks on its central points of failure, which lead to its ultimate demise. Its successors have learned this lesson by avoiding the central server altogether, regardless of the compromises needed to be made to achieve this.
- Cornel University - p2p lecture notes
- Indian Institute of Technology - p2p lecture notes
- WinMX OpenNap protocol spec