DevOps, Netstat Unix/Linux

UNIX / Linux: 10 Netstat

Cheetsheet:

List all ports using netstat -a

List all tcp ports using netstat -at

List all udp ports using netstat -au

List only listening ports using netstat -l

List only listening TCP Ports using netstat -lt

List only listening UDP Ports using netstat -lu

List only the listening UNIX Ports using netstat -lx

Show statistics for all ports using netstat -s

Show statistics for TCP netstat -st

Show statistics for UDP netstat -su

Add PID/Program Name to output netstat -ltp

Don’t resolve host, port and user name in netstat output netstat -ltpn

netstat will print information continuously every few seconds netstat -c

Display the kernel routing information using netstat -r

Find out on which port a program is running sudo netstat -ap | grep ssh

Find out which process is using a particular port netstat -an|grep 80

Show the list of network interfaces netstat -i

Display extended information on the interfaces netstat -ie

Netstat command displays various network related information such as network connections, routing tables, interface statistics, masquerade connections, multicast memberships etc.,

List All Ports (both listening and non listening ports)

List all ports using netstat -a

Output:

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 localhost:6942 *:* LISTEN

tcp 0 0 *:58882 *:* LISTEN

tcp 0 0 *:9191 *:* LISTEN

tcp 0 0 localhost:27017 *:* LISTEN

tcp 0 0 localhost:mysql *:* LISTEN

tcp 0 0 *:9292 *:* LISTEN

tcp 0 0 localhost:63342 *:* LISTEN

tcp 0 0 *:http *:* LISTEN

tcp 0 0 192.168.1.115:42722 server-xx-xx-xxx-x:http ESTABLISHED

List all tcp ports using netstat -at

Output:

tcp 0 0 192.168.1.115:45966 kul06s14-in-f195.:https ESTABLISHED

tcp 0 0 192.168.1.115:46602 maa03s22-in-f14.1:https ESTABLISHED

tcp 0 0 192.168.1.115:42204 maa03s21-in-f68.1:https ESTABLISHED

tcp 0 0 192.168.1.115:58846 maa03s21-in-f67.1:https ESTABLISHED

tcp 0 0 192.168.1.115:47272 maa03s22-in-f14.1:https ESTABLISHED

tcp 0 0 192.168.1.115:59862 maa03s22-in-f6.1e:https ESTABLISHED

tcp 0 0 192.168.1.115:60154 maa03s23-in-f3.1e:https ESTABLISHED

tcp 0 0 192.168.1.115:47182 maa03s22-in-f14.1:https ESTABLISHED

tcp 0 0 192.168.1.115:44704 a23-211-219-225.d:https ESTABLISHED

tcp 0 0 192.168.1.115:49298 bom05s08-in-f2.1e:https ESTABLISHED

List all udp ports using netstat -au

Output:

udp 0 0 *:54159 *:*

udp 0 0 *:mdns *:*

udp 0 0 *:mdns *:*

udp 0 0 *:mdns *:*

udp 0 0 *:58882 *:*

List only listening ports using netstat -l

Output:

unix 2 [ ACC ] STREAM LISTENING 21383 @/tmp/.ICE-unix/1520

unix 2 [ ACC ] STREAM LISTENING 22950 /run/user/1000/systemd/private

unix 2 [ ACC ] SEQPACKET LISTENING 11529 /run/udev/control

unix 2 [ ACC ] STREAM LISTENING 20420 /run/user/1000/keyring/control

unix 2 [ ACC ] STREAM LISTENING 21185 /run/user/1000/keyring/pkcs11

unix 2 [ ACC ] STREAM LISTENING 19715 /tmp/.X11-unix/X0

List only listening TCP Ports using netstat -lt

Output:

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 localhost:6942 *:* LISTEN

tcp 0 0 *:58882 *:* LISTEN

tcp 0 0 *:9191 *:* LISTEN

tcp 0 0 localhost:27017 *:* LISTEN

tcp 0 0 localhost:mysql *:* LISTEN

tcp 0 0 *:9292 *:* LISTEN

tcp 0 0 localhost:63342 *:* LISTEN

tcp 0 0 *:http *:* LISTEN

List only listening UDP Ports using netstat -lu

Output:

Proto Recv-Q Send-Q Local Address Foreign Address State

udp 0 0 *:54159 *:*

udp 0 0 *:mdns *:*

udp 0 0 *:mdns *:*

udp 0 0 *:mdns *:*

udp 0 0 *:58882 *:*

udp 0 0 localhost:59209 *:*

List only the listening UNIX Ports using netstat -lx

Output:

Proto RefCnt Flags Type State I-Node Path

unix 2 [ ACC ] STREAM LISTENING 21383 @/tmp/.ICE-unix/1520

unix 2 [ ACC ] STREAM LISTENING 22950 /run/user/1000/systemd/private

unix 2 [ ACC ] SEQPACKET LISTENING 11529 /run/udev/control

unix 2 [ ACC ] STREAM LISTENING 20420 /run/user/1000/keyring/control

unix 2 [ ACC ] STREAM LISTENING 21185 /run/user/1000/keyring/pkcs11

unix 2 [ ACC ] STREAM LISTENING 19715 /tmp/.X11-unix/X0

unix 2 [ ACC ] STREAM LISTENING 21186 /run/user/1000/keyring/ssh

unix 2 [ ACC ] STREAM LISTENING 21384 /tmp/.ICE-unix/1520

unix 2 [ ACC ] STREAM LISTENING 23734 /run/user/1000/pulse/native

Show the statistics for each protocol

Show statistics for all ports using netstat -s

Output:

Ip:

258229 total packets received

23 with invalid addresses

0 forwarded

0 incoming packets discarded

258083 incoming packets delivered

228944 requests sent out

124 outgoing packets dropped

2 dropped because of missing route

Show statistics for TCP netstat -st

Output:

IcmpMsg:

InType3: 433

InType11: 38

OutType3: 466

Tcp:

7352 active connections openings

98 passive connection openings

125 failed connection attempts

1396 connection resets received

18 connections established

211357 segments received

190589 segments send out

2857 segments retransmited

355 bad segments received.

5957 resets sent

Show statistics for UDP netstat -su

Output:

IcmpMsg:

InType3: 433

InType11: 38

OutType3: 466

Udp:

37594 packets received

466 packets to unknown port received.

0 packet receive errors

35126 packets sent

IgnoredMulti: 11474

Display PID and program names in netstat output using netstat -p

# netstat -ltp

netstat -p option can be combined with any other netstat option. This will add the “PID/Program Name” to the netstat output. This is very useful while debugging to identify which program is running on a particular port.

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 localhost:6942 *:* LISTEN 3329/java

tcp 0 0 *:58882 *:* LISTEN 1689/skype

tcp 0 0 *:9191 *:* LISTEN -

tcp 0 0 localhost:27017 *:* LISTEN -

tcp 0 0 localhost:mysql *:* LISTEN -

Don’t resolve host, port and user name in netstat output

# netstat -ltpn

When you don’t want the name of the host, port or user to be displayed, use netstat -n option. This will display in numbers, instead of resolving the host name, port name, user name.

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 127.0.0.1:6942 0.0.0.0:* LISTEN 3329/java

tcp 0 0 0.0.0.0:58882 0.0.0.0:* LISTEN 1689/skype

tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -

tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:63342 0.0.0.0:* LISTEN 3329/java

Print netstat information continuously

netstat will print information continuously every few seconds.

# netstat -c

Find the non supportive Address families in your system

# netstat --verbose

At the end, you will have something like this.

unix 2 [ ] STREAM CONNECTED 39668

unix 3 [ ] STREAM CONNECTED 21486 @/tmp/.X11-unix/X0

unix 3 [ ] STREAM CONNECTED 21353

unix 3 [ ] STREAM CONNECTED 21292

netstat: no support for `AF IPX' on this system.

netstat: no support for `AF AX25' on this system.

netstat: no support for `AF X25' on this system.

netstat: no support for `AF NETROM' on this system.

Display the kernel routing information using netstat -r

# netstat -r

Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt Iface

default 192.168.1.1 0.0.0.0 UG 0 0 0 wlp4s0

link-local * 255.255.0.0 U 0 0 0 wlp4s0

192.168.1.0 * 255.255.255.0 U 0 0 0 wlp4s0

Find out on which port a program is running

# sudo netstat -ap | grep ssh

unix 2 [ ACC ] STREAM LISTENING 21186 1255/gnome-keyring- /run/user/1000/keyring/ssh

Find out which process is using a particular port:

# netstat -an|grep 80

Show the list of network interfaces

# netstat -i

Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp0s25 1500 0 0 0 0 0 0 0 0 0 BMU

lo 65536 0 24768 0 0 0 24768 0 0 0 LRU

wlp4s0 1500 0 307452 0 0 0 207356 0 0 0 BMRU

Display extended information on the interfaces (similar to ifconfig) using netstat -ie:

# netstat -ie

enp0s25 Link encap:Ethernet HWaddr 50:7b:9d:70:4c:15

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Interrupt:20 Memory:e1200000-e1220000

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:24787 errors:0 dropped:0 overruns:0 frame:0

Non-SQL, MongoDB

What is NoSQL?

A NoSQL database environment is, simply put, a non-relational and largely distributed database system that enables rapid, ad-hoc organization and analysis of extremely high-volume, disparate data types. NoSQL databases are sometimes referred to as cloud databases, non-relational databases, Big Data databases and a myriad of other terms and were developed in response to the sheer volume of data being generated, stored and analyzed by modern users (user-generated data) and their applications (machine-generated data).

In general, NoSQL databases have become the first alternative to relational databases, with scalability, availability, and fault tolerance being key deciding factors. They go well beyond the more widely understood legacy, relational databases (such as Oracle, SQL Server and DB2 databases) in satisfying the needs of today’s modern business applications. A very flexible and schema-less data model, horizontal scalability, distributed architectures, and the use of languages and interfaces that are “not only” SQL typically characterize this technology.

From a business standpoint, considering a NoSQL or ‘Big Data’ environment has been shown to provide a clear competitive advantage in numerous industries. In the ‘age of data’, this is compelling information as a great saying about the importance of data is summed up with the following “if your data isn’t growing then neither is your business”.

Types of NoSQL Databases

There are four general types of NoSQL databases, each with their own specific attributes:

• Graph database – Based on graph theory, these databases are designed for data whose relations are well represented as a graph and has elements which are interconnected, with an undetermined number of relations between them. Examples include: Neo4j and Titan.
• Key-Value store – we start with this type of database because these are some of the least complex NoSQL options. These databases are designed for storing data in a schema-less way. In a key-value store, all of the data within consists of an indexed key and a value, hence the name. Examples of this type of database include: Cassandra, DyanmoDB, Azure Table Storage (ATS), Riak, BerkeleyDB.
• Column store – (also known as wide-column stores) instead of storing data in rows, these databases are designed for storing data tables as sections of columns of data, rather than as rows of data. While this simple description sounds like the inverse of a standard database, wide-column stores offer very high performance and a highly scalable architecture. Examples include: HBase, BigTable and HyperTable.
• Document database – expands on the basic idea of key-value stores where “documents” contain more complex in that they contain data and each document is assigned a unique key, which is used to retrieve the document. These are designed for storing, retrieving, and managing document-oriented information, also known as semi-structured data. Examples include: MongoDB and CouchDB.

What is MongoDb?

MongoDb is a Open Source database written in C++.

Drivers and client libraries are typically written in their respective languages, although some drivers use C extensions for better performance.

If the load increases, by adding more nodes (such as a computer), the performance can be retained.

It can be used to store data for very high performance applications (for example Foursquare is using it in production).

MongoDB does not support SQL It supports a rich, ad-hoc query language of its own.

MongoDb stores data as documents. So it is a document oriented database.

FirstName="Arun", Address="St. Xavier's Road", Spouse=[{Name:"Kiran"}], Children=[{Name:"Rihit", Age:8}].

FirstName="Sameer",Address="8 Gandhi Road".

Notice there are two different documents (separated by "."). Storing data in this fashion is called as document oriented database. MongoDb is a document oriented database.

MongoDB : Databases, Schemas and Tables

Databases : MongoDB is a document-oriented DBMS, with JSON-like objects comprising the data model, rather than RDBMS tables. MongoDB does not support joins nor transactions. However, it features secondary indexes, an expressive query language, atomic writes on a per-document level, and fully-consistent reads. MongoDB uses BSON, a binary object format similar to, but more expressive than JSON.

Schemas : MongoDB uses dynamic schemas. We can create collections without defining the structure, i.e. the fields or the types of their values, of the documents. You can change the structure of documents simply by adding new fields or deleting existing ones. Documents in a collection need unique set of fields.

Tables : MongoDB database stores its data in collections not in tables The collections are the rough equivalent of RDBMS tables. A collection holds one or more documents, which corresponds to a record or a row in a relational database table, and each document has one or more fields, which corresponds to a column in a relational database table.

MongoDB and ACID transactions

MongoDB does not support multi-document transactions, but provides atomic operations on a single document. Often these document-level atomic operations are sufficient to solve problems that would require ACID transactions in a relational database.

In MongoDB, you can embed related data in nested arrays or nested documents within a single document and update the entire document in a single atomic operation. Relational databases might represent the same kind of data with multiple tables and rows, which would require transaction support to update the data atomically.

CRUD operations create, read, update, and delete documents.

Create Operations

Create or insert operations add new documents to a collection. If the collection does not currently exist, insert operations will create the collection.

MongoDB provides the following methods to insert documents into a collection:

• db.collection.insert()
• db.collection.insertOne() New in version 3.2
• db.collection.insertMany() New in version 3.2

Ex: db.users.insert({name:”sue”, age:26, status: “A”})

Read Operations

Read operations retrieves documents from a collection; i.e. queries a collection for documents. MongoDB provides the following methods to read documents from a collection:

• db.collection.find()

Ex: db.users.find({age: {$gt:18}}, {name:1, addess:1}).limit(5)

More about filter & criteria

Update Operations

Update operations modify existing documents in a collection. MongoDB provides the following methods to update documents of a collection:

• db.collection.update()
• db.collection.updateOne() New in version 3.2
• db.collection.updateMany() New in version 3.2
• db.collection.replaceOne() New in version 3.2

In MongoDB, update operations target a single collection. All write operations in MongoDB are atomic on the level of a single document.

You can specify criteria, or filters, that identify the documents to update. These filters use the same syntax as read operations.

Ex: db.users.update({age: {$gt:18 }}, {$set: {status: “D”}}, {multi:true})

More about Update

Delete Operations

Delete operations remove documents from a collection. MongoDB provides the following methods to delete documents of a collection:

• db.collection.remove()
• db.collection.deleteOne() New in version 3.2
• db.collection.deleteMany() New in version 3.2

In MongoDB, delete operations target a single collection. All write operations in MongoDB are atomic on the level of a single document.

You can specify criteria, or filters, that identify the documents to remove. These filters use the same syntax as read operations.

Ex: db.users.delete({status: “D”})

More about Delete

How Big Is A Petabyte, Exabyte and Zettabyte ?

This is an intuitive look at large data sizes By Julian Bunn in Globally Interconnected Object Databases.

Bytes(8 Bits)

• 0.1 bytes: A binary decision
• 1 byte: A single character
• 10 bytes: A single word
• 100 bytes: A telegram OR A punched card

Kilobyte (1000 Bytes)

• 1 Kilobyte: A very short story
• 2 Kilobytes: A Typewritten page
• 10 Kilobytes: An encyclopaedic page OR A deck of punched cards
• 50 Kilobytes: A compressed document image page
• 100 Kilobytes: A low-resolution photograph
• 200 Kilobytes: A box of punched cards
• 500 Kilobytes: A very heavy box of punched cards

Megabyte (1 000 000 Bytes)

• 1 Megabyte: A small novel OR A 3.5 inch floppy disk
• 2 Megabytes: A high resolution photograph
• 5 Megabytes: The complete works of Shakespeare OR 30 seconds of TV-quality video
• 10 Megabytes: A minute of high-fidelity sound OR A digital chest X-ray
• 20 Megabytes: A box of floppy disks
• 50 Megabytes: A digital mammogram
• 100 Megabytes: 1 meter of shelved books OR A two-volume encyclopaedic book
• 200 Megabytes: A reel of 9-track tape OR An IBM 3480 cartridge tape
• 500 Megabytes: A CD-ROM OR The hard disk of a PC

Gigabyte (1 000 000 000 Bytes)

• 1 Gigabyte: A pickup truck filled with paper OR A symphony in high-fidelity sound OR A movie at TV quality
• 2 Gigabytes: 20 meters of shelved books OR A stack of 9-track tapes
• 5 Gigabytes: An 8mm Exabyte tape
• 20 Gigabytes: A good collection of the works of Beethoven OR 5 Exabyte tapes OR A VHS tape used for digital data
• 50 Gigabytes: A floor of books OR Hundreds of 9-track tapes
• 100 Gigabytes: A floor of academic journals OR A large ID-1 digital tape
• 200 Gigabytes: 50 Exabyte tapes

Terabyte (1 000 000 000 000 Bytes)

• 1 Terabyte: An automated tape robot OR All the X-ray films in a large technological hospital OR 50000 trees made into paper and printed OR Daily rate of EOS data (1998)
• 2 Terabytes: An academic research library OR A cabinet full of Exabyte tapes
• 10 Terabytes: The printed collection of the US Library of Congress
• 50 Terabytes: The contents of a large Mass Storage System

Petabyte (1 000 000 000 000 000 Bytes)

• 1 Petabyte: 5 years of EOS data (at 46 mbps)
• 2 Petabytes: All US academic research libraries
• 20 Petabytes: Production of hard-disk drives in 1995
• 200 Petabytes: All printed material OR Production of digital magnetic tape in 1995

Exabyte (1 000 000 000 000 000 000 Bytes)

• 5 Exabytes: All words ever spoken by human beings.
• From wikipedia:
  • The world's technological capacity to store information grew from 2.6 (optimally compressed) exabytes in 1986 to 15.8 in 1993, over 54.5 in 2000, and to 295 (optimally compressed) exabytes in 2007. This is equivalent to less than one 730-MB CD-ROM per person in 1986 (539 MB per person), roughly 4 CD-ROM per person of 1993, 12 CD-ROM per person in the year 2000, and almost 61 CD-ROM per person in 2007. Piling up the imagined 404 billion CD-ROM from 2007 would create a stack from the earth to the moon and a quarter of this distance beyond (with 1.2 mm thickness per CD).

The world’s technological capacity to receive information through one-way broadcast networks was 432 exabytes of (optimally compressed) information in 1986, 715 (optimally compressed) exabytes in 1993, 1,200 (optimally compressed) exabytes in 2000, and 1,900 in 2007.

• • According to the CSIRO, in the next decade, astronomers expect to be processing 10 petabytes of data every hour from the Square Kilometre Array (SKA) telescope.[11] The array is thus expected to generate approximately one exabyte every four days of operation. According to IBM, the new SKA telescope initiative will generate over an exabyte of data every day. IBM is designing hardware to process this information.

Zettabyte (1 000 000 000 000 000 000 000 Bytes)

• From wikipedia:
• The world’s technological capacity to receive information through one-way broadcast networks was 0.432 zettabytes of (optimally compressed) information in 1986, 0.715 in 1993, 1.2 in 2000, and 1.9 (optimally compressed) zettabytes in 2007 (this is the informational equivalent to every person on earth receiving 174 newspapers per day).
• According to International Data Corporation, the total amount of global data is expected to grow to 2.7 zettabytes during 2012. This is 48% up from 2011.
• Mark Liberman calculated the storage requirements for all human speech ever spoken at 42 zettabytes if digitized as 16 kHz 16-bit audio. This was done in response to a popular expression that states "all words ever spoken by human beings" could be stored in approximately 5 exabytes of data (see exabyte for details). Liberman did "freely confess that maybe the authors [of the exabyte estimate] were thinking about text."
• Research from the University of Southern California reports that in 2007, humankind successfully sent 1.9 zettabytes of information through broadcast technology such as televisions and GPS.
• Research from the University of California, San Diego reports that in 2008, Americans consumed 3.6 zettabytes of information.

More Detail

SSH Tunneling Made Easy

I was surprised at how long it took me to find a good HOWTO on setting up a simple SSH tunnel that I wanted to write up this Quick-Tip.

Using OpenSSH on a Linux/Unix system you can tunnel all of the traffic from your local box to a remote box that you have an account on.

For example I tunnel all of my outbound E-mail traffic back to my personal server to avoid having to change SMTP servers, use SMTP-AUTH, etc. when I am behind firewalls. I find that hotel firewalls, wireless access points, and the other various NATing devices you end up behind while traveling often do not play nice.

To do this I use the following:

ssh -f user@personal-server.com -L 2000:person-server.com:25 -N

The -f tells ssh to go into the background just before it executes the command. This is followed by the username and server you are logging into. The -L 2000:personal-server.com:25 is in the form of -L local-port:host:remote-port. Finally the -N instructs OpenSSH to not execute a command on the remote system.

This essentially forwards the local port 2000 to port 25 on personal-server.com over, with nice benefit of being encrypted. I then simply point my E-mail client to use localhost:2000 as the SMTP server and we're off to the races.

Another useful feature of port forwarding is for getting around pesky firewall restrictions. For example, a firewall I was behind recently did not allow outbound Jabber protocol traffic to talk.google.com. With this command:

ssh -f -L 3000:talk.google.com:5222 home -N

I was able to send my Google Talk traffic encrypted through the firewall back to my server at home and then out to Google. 'home' here is just an SSH alias to my server at home. All I had to do was reconfigure my Jabber client to use localhost as the server and the port 3000 that I had configured.

Hopefully this helps you to better understand SSH tunneling. If you found this page useful, you may also be interested in how to make your SSH connections faster.

More Detail

OpenSSH Server on Ubuntu

Introduction

This section of the Ubuntu Server Guide introduces a powerful collection of tools for the remote control of, and transfer of data between, networked computers called OpenSSH. You will also learn about some of the configuration settings possible with the OpenSSH server application and how to change them on your Ubuntu system.

OpenSSH is a freely available version of the Secure Shell (SSH) protocol family of tools for remotely controlling, or transferring files between, computers. Traditional tools used to accomplish these functions, such as telnet or rcp, are insecure and transmit the user's password in cleartext when used. OpenSSH provides a server daemon and client tools to facilitate secure, encrypted remote control and file transfer operations, effectively replacing the legacy tools.

The OpenSSH server component, sshd, listens continuously for client connections from any of the client tools. When a connection request occurs, sshd sets up the correct connection depending on the type of client tool connecting. For example, if the remote computer is connecting with the ssh client application, the OpenSSH server sets up a remote control session after authentication. If a remote user connects to an OpenSSH server with scp, the OpenSSH server daemon initiates a secure copy of files between the server and client after authentication. OpenSSH can use many authentication methods, including plain password, public key, and Kerberos tickets.

Installation

Installation of the OpenSSH client and server applications is simple. To install the OpenSSH client applications on your Ubuntu system, use this command at a terminal prompt:

sudo apt-get install openssh-client

To install the OpenSSH server application, and related support files, use this command at a terminal prompt:

sudo apt-get install openssh-server

The openssh-server package can also be selected to install during the Server Edition installation process.

Generate Key

sudo-keygen -t rsa

More