CWP Server Root Compromise via External API Port 2304 – Real Incident Analysis

CWP Server Root Compromise via External API Port 2304 – Real Incident Analysis and Recovery


Introduction

I recently investigated a serious security incident on a production CWP7 server running AlmaLinux.

What initially looked like a compromised Joomla website eventually turned out to be something much more serious: the entire server had obtained unauthorized root-level access.

The investigation uncovered:

This article documents the investigation, indicators of compromise, containment steps and lessons learned.

Important: This article is written from a defensive system-administration perspective. It intentionally does not reproduce the malicious exploit payload.

---

1. How the Incident Was First Detected

The investigation started after unexpected changes were detected inside one hosted Joomla website.

Files such as:



appeared or changed unexpectedly.

One website also started serving gambling/SEO spam related to "LAMTOTO", while another site contained a similar malicious page.

At first, the obvious suspicion was:



However, the timestamps did not fit that theory.

Files belonging to unrelated hosting accounts were being modified by:



That immediately changed the scope of the investigation.

This was no longer a single compromised website.

The server itself had to be considered compromised.

---

2. Cross-Account Activity

A particularly useful indicator was a tiny file named:



The same file appeared across numerous unrelated hosting accounts.

Its content was identical:



More importantly, web-server logs showed requests for this file coming from the server's own IPv4 and IPv6 addresses using:



This strongly indicated that a local script running on the server was creating or testing files across multiple hosting accounts.

That is not normal website-level malware behavior.

A compromised Joomla account normally cannot freely write into unrelated customers' home directories.

Root access can.

---

3. The CWP External API Became the Main Suspect

CWP uses TCP port:



for its External API.

The API configuration on the affected server contained:



and forwarded PHP requests to:



The associated PHP-FPM pool configuration was then checked.

The result was critical:



Running processes confirmed it:



This means that code successfully executed through this privileged CWP API path can execute with root privileges.

---

4. Suspicious Requests to /v1/backup

The CWP API access log showed repeated requests from:



against:



The timeline included:



The final August responses were considerably larger than the earlier responses.

CWP's debug log showed that:



was rewritten internally to:



and handled by CWP's API application.

---

5. The Smoking Gun: CWP API Log

The most important evidence was eventually found in:



Entries associated with the backup API contained attacker-supplied shell execution.

The log clearly showed a backup operation followed by:



and subsequent communication with infrastructure associated with:



The injected code performed filesystem/domain reconnaissance and contacted an external HTTP service.

At this point there was little ambiguity left.

The compromise chain was essentially:



The earliest confirmed successful execution discovered during the investigation was:



The later August activity was therefore not the original compromise.

It was continued access to an already vulnerable server.

---

6. Root Persistence Through GSocket

Root shell history contained several highly suspicious commands involving:



One malicious binary was installed as:



This is particularly deceptive because:



is normally an SSH private key.

But:



was an executable binary.

Its SHA256 was:



The binary communicated with:



and was associated with GSocket/gs-netcat functionality.

---

7. Systemd Persistence Disguised as a Kernel Process

The attacker created a systemd service configured approximately as:



The executable was launched using a fake process name:



This was clever because Linux systems can legitimately contain a kernel workqueue named:



The difference is critical.

The malicious process had a userspace executable:



A genuine kernel thread does not point to a normal userspace executable through:



The service automatically restarted approximately five minutes after being killed because it had:



Simply killing the process therefore did not remove the backdoor.

The systemd persistence had to be stopped and masked.

---

8. Additional Suspicious System Files

Three additional files appeared during the compromise window:



The most significant was:



Characteristics included:



The old modification timestamp was clearly inconsistent with its filesystem birth time.

This is a common anti-forensic technique called timestamp manipulation or timestomping.

The file was preserved for evidence and then removed from its active system location.

---

9. Website-Level Damage

After obtaining root access, the attacker affected several hosted websites.

Observed behavior included:

• Immutable public_html directories

One particularly dangerous lesson was that a website could appear repaired while malicious content was still being served from Nginx FastCGI cache.

After restoring clean website files, one domain continued serving the malicious page.

The filesystem looked clean.

A unique query parameter showed a clean page.

The normal URL showed the old malicious page.

The reason was stale Nginx cache.

Purging:



resolved the stale output.

Therefore:

Never assume that restoring filesystem files automatically removes cached malicious pages.

---

10. Immutable Webroot Attributes

Some public_html directories had the immutable filesystem attribute:



This can be checked using:



Example:



An immutable directory cannot be modified normally, even by the hosting user.

During incident response this can cause confusing symptoms such as:

To remove an unexpected immutable flag:



However, do not automatically remove immutable flags everywhere.

Some administrators deliberately use them as a protection mechanism.

Check timestamps and history first.

---

11. Immediate Containment

The malicious systemd service was:



The executable was preserved for forensic purposes and removed from:



The attacker's C2 IP was blocked in CSF.

The suspected CWP attacker IP was also blocked:



Most importantly, CWP External API port 2304 was removed from:



in:



Then CSF was reloaded:



Verification:



There should be no rule similar to:



CWP itself may continue listening locally:



That does not automatically mean it is reachable from the Internet.

The firewall is what determines external reachability.

---

12. Recommended CWP Configuration

If you do not actively use:

• remote automation requiring the External API

there is little reason to expose:



to the entire Internet.

A much safer architecture is:



If the API is genuinely required, restrict access to specific trusted IP addresses rather than:

---

13. Useful Detection Commands

Check port 2304:



Check CWP API access:



Check suspicious CWP API execution:



Check GSocket indicators:



Check known C2 connections:



Check suspicious files:



Check suspicious systemd references:



Check immutable webroots:



Check recent root-owned website files:

---

14. Indicators of Compromise

Indicators found during this incident included:



Indicators should never be treated individually as automatic proof of compromise.

Context matters.

For example, a historical log containing an IOC after cleanup does not mean the malicious process is currently active.

---

15. Security Audit Script

After the incident I created a read-only audit script that can be deployed across CWP/Linux servers.

The audit checks:

An important design decision was:

The audit detects and reports. It does not automatically delete or repair anything.

Automatic malware removal on a production server can easily destroy legitimate data or forensic evidence.

---

16. SPF / DKIM / DMARC and Security Reports

Another problem discovered during the investigation was that security audit emails were not arriving in Gmail.

Postfix showed:



The server's hostname had no suitable SPF record and DKIM was not published.

For the server hostname, an SPF record was added similar to:



After DNS propagation:



should return the SPF record.

If DMARC is:



mail authentication must be configured correctly or Gmail may reject automated security reports.

---

17. Important Lesson About Root Passwords

Changing the root password is important after compromise.

However:

Changing the root password does not fix an unauthenticated remote code execution vulnerability.

If the attacker entered through a public CWP API vulnerability, they never needed the root password.

Therefore the correct order is:



Passwords are only one part of the recovery process.

---

18. Can the Existing Server Be Trusted Again?

This is the most important conclusion.

Once an attacker has obtained unrestricted root access, it is impossible to prove with certainty that every modification has been discovered.

Root can alter:

Removing the known malware is therefore containment, not restoration of trust.

The safest recovery procedure after confirmed root compromise remains:

---

19. Major Lessons Learned

The biggest lessons from this incident were:

• Do not assume the visible compromised website is the entry point.
  
  The affected Joomla websites were consequences of the compromise, not necessarily the cause.
  
• Always correlate filesystem timestamps with web/API logs.
  
  This was how the CWP API activity was linked to subsequent server-side actions.
  
• Control-panel APIs deserve the same protection as SSH.
  
  A privileged API publicly exposed on the Internet can represent a much larger risk than an individual CMS.
  
• Running administrative PHP as root greatly increases the impact of an application vulnerability.
  
  The CWP cwpsrv PHP-FPM pool was running as root.
  
• Killing a malware process is not enough.
  
  Systemd restarted the malicious process automatically.
  
• Check caches after restoring a compromised website.
  
  Nginx may continue serving previously cached malicious content.
  
• Inspect all hosting accounts after server-level compromise.
  
  The attacker had access across account boundaries.
  
• Do not rely solely on malware scanners.
  
  Timeline reconstruction, logs, process ancestry, filesystem metadata and network connections were critical.
  
• Keep management interfaces restricted whenever possible.
  
  If an API is not required publicly, do not expose it publicly.

---

20. Final Conclusion

What initially appeared to be a Joomla SEO-spam infection turned out to be a complete server compromise.

The key evidence showed:



The incident demonstrates why server administrators should never limit an investigation to the website where the first symptoms appear.

A compromised site can be the victim of a much deeper server-level intrusion.

If you operate CWP servers, I strongly recommend reviewing whether External API port 2304 needs to be exposed at all, auditing /var/log/cwp/cwp_api.log, reviewing privileged PHP-FPM pools, monitoring unexpected root-owned website files and maintaining off-server clean backups.

And most importantly: if root compromise is confirmed, cleaning visible malware should be considered temporary containment — not proof that the server is trustworthy again.