Skip to main content
Star us on GitHub Star

Troubleshooting

Increase Log Level

Set the log level to DEBUG to identify the activity that is occurring at the same time as the problem.

# set the logLevel to "debug" in /var/lib/ziti/config.json
sudo -u ziti ziti-edge-tunnel set_log_level --loglevel DEBUG

The tunneler obeys the value of logLevel in /var/lib/ziti/config.json. The initial value may be set with run --verbose 4, but setting this option on subsequent runs has no effect on log level.

Capture the Current Process Log

journalctl _SYSTEMD_INVOCATION_ID=$(systemctl show -p InvocationID --value ziti-edge-tunnel.service) -l \
| tee /tmp/ziti-edge-tunnel.log

Systemd Service Won't Start or Keeps Restarting

Reload the systemd service unit definitions to rule out a stale definition.

sudo systemctl daemon-reload

Inspect the service unit.

sudo systemctl cat ziti-edge-tunnel.service

Check the service status for an error message.

sudo systemctl status ziti-edge-tunnel.service

Monitor the service logs.

sudo journalctl -u ziti-edge-tunnel.service

Intercepting or Hosting Not Working

Inspect the identity and router info for a running tunneler process. This creates a file named like {{identity name}}.ziti for each loaded identity. Each file summarizes the available services and router connections for the identity.

sudo -u ziti ziti-edge-tunnel dump -p /tmp/ziti-dump-dir/

Find tunneler's nameserver IP.

$ resolvectl --interface=ziti0 dns
Link 19 (tun0): 100.64.0.2

Query the Ziti nameserver to find the intercept IP address for a service.

dig +noall +answer my.ziti.service.example.com @100.64.0.2
Output
my.ziti.service.example.com. 60 IN    A       100.64.0.3

The tunneler provides end-to-end TCP handshake. Test the service's ability to accept connections even if it does not provide a greeting or banner as shown in the OpenSSH server example below.

# wait up to 3 seconds for a TCP handshake on port 443
$ ncat -vzw3 100.64.0.3 443
Ncat: Connected to 100.64.0.3:443.
Ncat: 0 bytes sent, 0 bytes received in 0.08 seconds.
# wait up to 3 seconds for an OpenSSH server greeting on port 22
$ ncat -vw3 100.64.0.3 22
SSH-2.0-OpenSSH_7.4

Process Keeps Crashing

A crash may be caused by a segmentation fault. If saving a Corefile is enabled, Linux will create a core dump file according to this pattern file: /proc/sys/kernel/core_pattern. Ubuntu configures this to use Apport. Read more about core dumps.

Please raise a GitHub issue if the tunneler crashes.

Operation Not Permitted

Delete the /tmp/.ziti directory and restart the tunneler to solve this issue.

The tunneler may log a warning about an operation not permitted and failure to start the socket server if the hard-coded path /tmp/.ziti is not owned by the run-as user. This can happen if you first run the tunneler as root and then as a non-root user.

WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1686 make_socket_path() failed to set ownership of /tmp/.ziti to 1003:1003: Operation not permitted (errno=1)
WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1730 run_tunneler_loop() One or more socket servers did not properly start.

Another symptom of the same issue is this error when attempting to send a message to the IPC socket server, which is not running.

failed to connect: -111/connection refused