Coinapult has halted its services after a compromise of its hot wallet lead to the theft of 150 BTC. Coinapult notified its customers via twitter to cease sending bitcoin to existing Coinapult addresses at this time. A notice on Coinapult's website currently reads:

Coinapult is investigating a compromise of its hot wallet.

Customers should NOT SEND bitcoin to existing Coinapult addresses starting IMMEDIATELY. This includes Lock Addresses (automatic lock address).

Coinapult has stopped services during investigation. Withdrawals are currently disabled. It appears 150 bitcoins were stolen. The majority of funds were not in the hot wallet.

Updates will be provided as they become available.

Update – Coinapult has released an incident report detailing the theft. It is available here and is mirrored below for posterity.

March 17 2015 Incidence Report

Summary

At 9:27 UTC an unauthorized withdrawal for 150 BTC was sent from Coinapult's hot wallet to this address: 12LszeXACdj9bdETzv8BkXyWeabZ1151aA

The transaction id is:

a050ffade5baac2c58dc17db27a1b1a12f7efcdafc3c41f559f0e1bb7177f474

As of 15:41 UTC these coins have not been spent.

Coinapult Team Involved

Ira – CEO with physical and SSH access to servers

Zach – IT Admin with physical and SSH access to servers

GP – CTO with SSH access to servers

Cindy – Developer and forensics expert with SSH access to servers

Justin – COO with no access to servers

Robinson – Customer service with no access to servers

There are 2 people with physical access to the affected servers. The servers are kept in a tier 3 data center with layers of physical security.

There are 4 people with SSH keys to the production finance server where the hot wallet is kept. To connect to the finance server, one of these would have to log into the company VPN, then use their SSH keys to log in to the server via command line.

Each of the production key holders' laptops were inspected by the others for network activity from the time window with nothing suspicious found, however Zach's laptop was exhibiting strange behavior reminiscent of a MITM attack. While we were all using the same local network, his IP address showed Gabon while the others all showed Panama. After we discovered this, Zach powered down his laptop and we are taking the hard drive out for forensic analysis.

Suspicious Timing

On Friday March 13th two things happened which are potentially related to this incident.

1. The data center where the finance server is hosted experiences an all day outage. The Panamanian government websites and numerous other local businesses were also down. The phones at the data center were down. Zach logged in to almost every server in the data center that day, as part of our recovery from the outage.

2. GP created and emailed Justin, Ira, and Zach a plan to transition all IT services to different servers outside of the data center. This may have been a last chance notice for the attacker, as their penetration work would be undone in the transfer.

In addition to the March 13th events, the previous 2 weeks have been unusually problematic for system stability. We have had hard drive problems, CPU problems, and all sorts of side effects. While the causes of most of these are known, it is possible some of it was malicious and/or masking other activity.

Clues from Finance server

1.

/var/log/auth.log appears to be modified. There is a blank line around 10:24-10:27 UTC and auth.log.1 is empty. Auth.log.1 should be full of data from the past few days.

2.

/root/.bash_history was last modified on Mar 14 19:38 UTC. The last 4 entries are:

nano auth.log

nano syslog

nano ufw.log

ls

These commands appear to have been run by the attacker, as the pattern does not match Coinapult usage. It looks like he opened these key log files with the intent to doctor them after he leaves. The lack of other commands and evidence suggests the possibility of a root kit. This may show up in forensics analysis of the hard drives. The use of 'nano' is also odd, as most hackers would chose a different text editor.

Clues from API & Data servers

1.

API server takes 30 minutes to send a message to the transaction processing engine (on the data server) at around the time of the attack. This is what caused to reach out to Robinson at 9:15 UTC. The cause of this is currently unknown. Compromising either of these servers would not allow direct access to the hot wallet or allow Finance server access.

Clues from SaaS servers

Coinapult runs a second set of production servers to rent out on a SaaS model. These SaaS servers have the same access rights as Coinapult's own. (2 physical access, 4 virtual) At the time of the theft, the SaaS finance server had over 500 BTC in it, which were not touched. This indicates the hacker did not know about that wallet. Of course, the funds have been moved and are now safe.

Objective Timeline

All times below are UTC-5.

1:49 – Ira requests hot wallet top off with 100 BTC from Bitfinex.

2:36 – Ira logs in to VPN (according to his syslog)

2:36 – Ira logs in to Finance server (according to server log)

2:37 – Ira runs sendmany to split outputs for optimal sending performance during the night. This was unnecessary as the 100 BTC had not shown up yet, but Ira did not notice that. Sendmany transaction:

3b7e8ac67bf9597fb38a41d1b1354a3d06e9078dd0c191191bae344dc419ec1d

3:55 – Bitfinex sends 100 BTC withdrawal: 4691cf88a74cc752c3777ec9952500cc04bd02ea6f979e326d3550b1581723c3

4:15 – notifies Robinson about improperly canceled transactions

4:27 – Withdrawal by hacker is made:

a050ffade5baac2c58dc17db27a1b1a12f7efcdafc3c41f559f0e1bb7177f474

4:54 – Robinson sends out email about transactions being stalled and HW being suspiciously low

4:58 – Robinson calls Zach and Zach starts trying to connect to VPN (according to his syslog)

5:17 – Zach successfully logs in to VPN (according to his syslog)

5:22 – Zach logs in to Finance server (according to server log)

5:31 – Zach sends email saying processes are running but can't assess HW on his own

8:42 – Ira has done enough investigation to identify that there has been 150 BTC withdrawn to an unknown address. Emails this info to the group.

9-12 Majority of funds are withdrawn from HW (minus change to watch). Customers (i.e. ) are notified and public notice is placed on our website. Team investigates and identifies the contents of this report.

Robinson's Account

Robinson was first on the scene, and so far, all evidence and corroborations agree with Robinson's timeline. Robinson had neither access to the servers nor the technical knowledge required to perpetrate the attack.

10:11:50 AM Ira hey, what exactly happened this morning?

10:11:53 AM can you give me the timeline please?

RO 10:12:15 AM Robinson sure.

10:13:02 AM sent a message about 2 transactions at 4:15 this morning

10:14:30 AM I saw the payments were canceling incorrectly, but I didn't know the reason

IR 10:14:56 AM Ira you were awake?

RO 10:15:33 AM Robinson yea, I woke up at 3, couldn't sleep, was doing some journaling when the message was received

10:16:25 AM the HW low balance emails started being sent at 4:30

10:17:30 AM I messaged both you and justin about that

10:18:16 AM justin pointed out that 100 BTC had been withdrawn from Bitfinex, but the volume in admin didn't add up to a low HW

10:18:31 AM that is when I sent the email and started calling you

IR 10:18:46 AM Ira justin was up to?

RO 10:18:46 AM Robinson after I tried you a couple of times I called zach

10:19:45 AM he said he was already awake

10:20:46 AM I spoke with Zach for ~2 minutes, mainly to tell him to check the email I sent

IR 10:23:09 AM Ira then he sent his email saying everything is fine?

RO 10:26:17 AM Robinson he said on the phone he would log in to investigate

10:26:48 AM he said in the email he doesn't know how to troubleshoot the hw

IR 10:27:22 AM Ira to clarify

10:27:28 AM Justin was already awake or Zach was?

RO 10:27:45 AM Robinson Justin said he was already awake.

10:28:15 AM When i called Zach, it sounded like I woke him up

Next Steps

1.

We have powered down and isolated all of the hardware in the data center. The plan is to disassemble them then run forensics on the hard drives to see if we can recover data from the manipulated logs or anywhere else.

2.

Zach is disassembling his laptop to run forensics on it.

3.

While we are moving the hardware out of the data center, we will ask for access logs and/or surveillance footage relevant to our situation. In addition, we will gather more information about the March 13th outage they experienced.