During our regular maintenance, after rebooted one SRX345, and found it stuck at db mode, which is debug mode.

After a short and quick analysis, I found Juniper JunOS devices may get stuck in the boot process or fail to boot the OS, in rare cases, after a sudden power loss or ungraceful power shut down. Juniper  routers, switches and firewalls  can experience file system corruption, which prevents the device from recovering to a functional state. It is recommended that customers minimize their log configurations to prevent excessive read/writes to the file system, which reduces the stress on storage media and reduces the potential occurrence of this issue. Moreover, if abrupt power failures are transient for a very short period of time, the availability of an UPS can also prevent the device from experiencing a sudden power loss.

You do not have to worry about damaging hardware in these situations, as the hardware cannot tell the difference between a graceful shutdown and pulling the power cord. The potential for damage is with the file system structure. It is possible for data to be corrupted, when the computer's power is interrupted with the operating system running. The data could be in the nodes, which could result in files being lost or file contents being corrupted.

Although rare, this issue more likely occurs on platforms that use a UNIX/BSD-based operating system, such as Junos, to access the flash-based storage media.

“Although rare, file system damage can occur with an abrupt power off, which may cause problems on the next boot. Use the request system halt or request system reboot command to gracefully shut down or reboot the OS. Once the OS is halted, it is safe to remove power.”  - from O'Reilly Media’s JUNOS Enterprise Switching book.

There are a couple of KB discussing the fix. KB29811 is using a USB to copy a snapshot from healthy device to faulty device. KB20046  suggest to press space to go to u-boot prompt and enter some commands to fix issue.

db> help
    DDB Quick Help  
  -------------------  
Type 'c' to continue, 'reset' or 'panic' to restart. 

print       p           examine     x           search      set         write       
w           delete      d           break       dwatch      watch       dhwatch     
hwatch      step        s           continue    c           until       next        
match       trace       alltrace    where       bt          call        show        
ps          gdb         reset       kill        watchdog    thread      panic       
ddbdumpsys  dumpsys     
db> reset

Issue Symptons:

• Normally, each firewall rule on the SRX auto-updates a snmp counter for hit-count, regardless of whether 'count' is configured or not.  Juniper Space Security Director periodically polls these OIDs and updates the hit-count.   
• In Junper Space 16.1 R1, the issue found is unable to view policy hit counts from Juniper Space Security Director, but SRX itself is keep updating. 

Actions Taken:

• Verify Security Appliance Policy Hits from Command line

Based on Juniper "Junos Space Virtual Appliance Installation and Configuration Guide" , JunOS Space " must deploy the virtual appliance on a VMware ESX, VMWare ESXi or KVM server, which provides a CPU, hard disk, RAM, and a network controller, but requires installation of an operating system and applications to become fully functional."

In my test environment, one JunOS Space has been installed on Citrix Xen environment and it is working fine until we tried to import a license.

The license was generated from Juniper License site and emailed to us in a txt file. It used to work on another machine hosted in Vmware ESX environment. Unfortunately, this time, JunOS Space said no.

The License Information windows says:
License upload failed. Please check the following:
1) License data format
2) License Keys

Juniper Space VE at Citrix Xen Server - License Error

My old post "Import Existing Juniper SRX Cluster into JunOS Space Security Director" was created based on Space 14.1 and SRX11.x version. Now both have been upgraded. Space NMP and Security Director have been upgrade to 16.1 (Post is here). SRX240H has been upgrade to 12.1D46.55.

Basically, all steps are similar except the web interface is different. What you need to do is to configure your SRX cluster with a master-only ip on both nodes. The configuration should looks like this:

Usually you can easily upgrade an application from the Junos Space user interface. You must download the image file for the new version of the application, navigate to the Applications page (Network Management Platform > Administration > Applications) and select the application that you want to upgrade. From the right-click menu, choose Upgrade Application to upload the image file into Junos Space via HTTP or SCP.

But upgrade JunOS Space to latest version 16.1 is different and it is not a easy task. There are many steps to follow especially the last step to upgrade to 16.1 from 15.2R2. Here is my recent upgrade procedures.

Steps to upgrade JunOS Space 14.1  to the latest version 16.1:

One of Juniper Firewall SRX240H had a serious crash. Manual reboot/shutdown did not work. To reset it, I would have to do a hard reset / power cycle device.

It would allow to log in from console, but you wont be able to see any configuration.
Here is outputs from this crashed Juniper SRX240H console:

Juniper SRX is a stateful firewall and allows traffic which matches an existing session. Sessions are created when a TCP SYN packet is received and it is permitted by the security policy. This of course means that the firewall needs to see both directions of a flow (client-server and server-client), otherwise these checks will block legitimate packets.

Following flow chart illustrates packet flow sequences both when SYN flag checking is enabled and when it is disabled.

By default, security TCP check is enabled on all TCP flow sessions. The Junos operating system (Junos OS) performs the following operations during TCP sessions:

• Checks for SYN flags in the first packet of a session and rejects any TCP segments with non- SYN flags that attempt to initiate a session.
• Validates the TCP sequence numbers during stateful inspection.

Reset packet is turned off for non-SYN session TCP packets:

{primary:node0} root@fw-mgmt-trn1-1> show security zones node0: -------------------------------------------------------------------------- Security zone: MGMT1   Send reset for non-SYN session TCP packets: Off   Policy configurable: Yes   Interfaces bound: 1   Interfaces:     reth4.201 Security zone: TSMGMT   Send reset for non-SYN session TCP packets: Off   Policy configurable: Yes   Interfaces bound: 1   Interfaces:     reth4.198 Security zone: MGMT2   Send reset for non-SYN session TCP packets: Off   Policy configurable: Yes   Interfaces bound: 1   Interfaces:     reth3.0

We can enable reset packets when received non-syn tcp session packets.

{primary:node0}[edit] root@fw-mgmt-1# set security zones security-zone MGMT1 tcp-rst {primary:node0}[edit] root@fw-mgmt-1# set security zones security-zone TSMGMT tcp-rst {primary:node0}[edit] root@fw-mgmt-1# set security zones security-zone MGMT tcp-rst

Check the settings again:

root@fw-mgmt-1> show security zones  node0: -------------------------------------------------------------------------- Security zone: MGMT1   Send reset for non-SYN session TCP packets: On   Policy configurable: Yes     Interfaces bound: 1   Interfaces:     reth4.201 Security zone: TSMGMT   Send reset for non-SYN session TCP packets: On   Policy configurable: Yes     Interfaces bound: 1   Interfaces:     reth4.198 Security zone: MGMT2   Send reset for non-SYN session TCP packets: On   Policy configurable: Yes     Interfaces bound: 1   Interfaces:reth3.0

Junos OS provides a mechanism for disabling security checks on TCP packets to ensure interoperability with hosts and devices with faulty TCP implementations. During no-SYN-check the Junos OS does not look for the TCP SYN packet for session creation. No-sequence check disables TCP sequence checking validation. Also, increases throughput. SYN check and sequence check are enabled by default. The set security flow command disables TCP SYN checks and TCP sequence checks on all TCP sessions thus reduces security. This may be required in scenarios with customers like big transfer files, or with applications that do not correctly work with standards.

Another reason to disable syn-check and sequence-check is the asymmetric flows in your environment. It is best, whenever possible, to ensure that asymmetric flows do not occur; but this is not always possible. So, you can disable these checks globally on the SRX device.

To disable TCP packet security checks:

set security flow tcp-session no-syn-check set security flow tcp-session no-sequence-check

After you disabled the tcp options, tcp-syn-check, and tcp-sequence-check that are configured at global level, you might want to configure TCP packet security checks at the policy level.

Note: Disabling the global SYN check and enforcing the SYN check after policy search will greatly impact the number of packets that the router can process. This in turn will result in intense CPU operations.

Configure the checking for the TCP SYN bit before creating a session:

[edit] user@host# set security policies from-zone Zone-A to-zone Zone-B policy pol1 then permit tcp-options syn-check-required

Configure the checking for sequence numbers in TCP segments during stateful inspection:

[edit] user@host# set security policies from-zone Zone-A to-zone Zone-B policy pol1 then permit tcp-options sequence-check-required

It is also possible to disable TCP SYN or sequence checking on one policy and enable them on all other policies, an apply-group can be used to complete this configuration based on KB24566.


Reference:

• [J-Series/SRX] Example: Configuring TCP SYN Check options on a per policy basis
• [ SRX ] : How to selectively disable TCP SYN or Sequence checking
• Understanding TCP SYN Checking - Juniper Networks
• Example: Disabling TCP Packet Security Checks for SRX Series Services Gateways

1.Understand Juniper SRX logging Type:

1.1 System Logging

SRX Series devices can send system log messages from the control plane (Routing Engine) to one or more destinations. Destinations can include local files on the SRX Series device (because the SRX Series device is a syslog server), remote syslog servers, user terminals, and the system console.

admin@fw-1> show configuration system syslog archive size 750k files 2; user * {     any emergency; } host 10.9.0.33 {     any any;     change-log none;     interactive-commands none;     explicit-priority; } host 10.9.8.52 {     any any;     source-address 10.9.8.20; } file messages {     any critical;     authorization info;     explicit-priority; } file interactive-commands {     interactive-commands error; } }

1.2 Traffic Logging (Event Mode)

You can use traffic logs to track usage patterns or troubleshoot issues for a specific policy. You can configure a policy so that traffic information is logged when a session begins (session-init) and/or closes (session-close). To generate traffic logs for multiple policies, you must configure each policy to log traffic information. You also must configure syslog messages with a severity level of info or any. In the default configuration, these messages and all other logging messages are sent to a local log file named messages.

admin@fw-1> show configuration system syslog archive size 750k files 2; user * {     any emergency; } host 10.9.0.33 {     any any;     change-log none;     interactive-commands none;     explicit-priority; } host 10.9.8.52 {     any any;     source-address 10.9.8.20; } file messages {     any critical;     authorization info;     explicit-priority; } file interactive-commands {     interactive-commands error; } file traffic-create {     any any;     match RT_FLOW_SESSION_CREATE;     structured-data; } file traffic-deny {     any any;     match RT_FLOW_SESSION_DENY; } file traffic-flow {     user info;     match RT_FLOW;     archive size 1000k files 5 world-readable;     structured-data; }

1.3 Notes:

	System Logging	Traffic Logging
SRX Branch Devices SRX100 SRX110 SRX210 SRX220 SRX240 SRX550 SRX650	KB16502	KB16509
SRX High-End Devices SRX1400 SRX3400 SRX3600 SRX5600 SRX5800	KB16502	KB16506

2. Understand Juniper SRX Logging Methods:

Control Plane and Data Plane

2.1 Control Plane Logging

 the control plane logs have to do with events triggered by daemons on the control plane. This includes messages about the underlying hardware (chassisd), general-purpose messages (messages), and various protocol daemons like IDPD, appidd, and so on. Control plane logging is on by default to log locally, but you can override this with your own logfiles, syslog hosts, and criteria for different log messages. All logs are stored in the /var/log directory on the control plane. It has been described at section 1.1

Services on the control plane:

• Management Daemon (MGD):  Provides the interface between the UI components and the backend configuration and is responsible for acting on the Junos configuration to the system itself.
• Routing Protocol Daemon (RPD) : All routing protocols including RIP, OSPF, IS-IS, BGP, PIM, IPv6 counterparts, and so on.
• User interfaces: Console, Telnet, SSH, J-Web, NetConf.
• Filesystem interfaces: FTP/SCP.
• Syslogd: Logging subsystem on the control plane, different than what is on the data plane. This generates the OS and application logs on the control plane.
• Networking services: DNS, DHCP, NTP, ICMP, ARP/ND, SNMP.
• Chassisd: Controls the hardware operations of the data plane and interfaces with the components to ensure they are active and operating properly.
• JSRPD: This is the high availability daemon that runs the HA functionality between two SRX chassis in an HA cluster.

2.2 Data Plane Logging

Data plane logs are primarily those generated by components that process traffic on the data plane. These include the firewall logs (RT_LOG, which stands for Real-Time Log because it is not stored on the data plane) from the flowd process, IPS logs, UTM logs, and logs from other security components like Screens. Data plane logging is off by default and must be configured. Typically, it is recommended that you send logs off the SRX to a syslog host due to the large volume of logs that can be generated from the data plane, particularly on high-end SRX platforms like the 5800. In fact, it can take an entire infrastructure of syslog servers to handle the large volume of syslog messages that the high-end SRX can generate per second. For this reason, there are two different mechanisms that we can use to log messages to the control plane, as discussed in the next section.

Services on the data plane:

• Intrusion Detection and Prevention Daemon (IDPD)
• IKED
• PKID

2.2.1 Event Mode 

Event mode  - control plane log processing - used on low end devices. Optionally even rate can be specified. Once event mode is enabled under "security" then the logging to local file can configure under "system syslog" as above at section 1.2.  You also can configure that security traffic logs are handled through the eventd process and sent with system logs though control panel Routing Engine.

admin@fw-1> show configuration security log {     mode event;     event-rate 1000;     format sd-syslog;     source-address 10.9.8.20;     stream securitylog {         format sd-syslog;         category all;         host {             10.9.8.52;         }     }     stream LogCollector {         host {             10.9.20.17;         }     }     stream TO-10.9.20.33 {         format sd-syslog;         category all;         host {             10.9.20.33;         }     } }

2.2.2 Stream Mode

Stream mode - data plane logging - Normally used on high end SRX devcies but can be configured on any SRX devices. Under security the syslog parameters can be specified, e.g. syslog server, syslog format, facility.

Note: SRX can only log to the control plane (Event mode) or log out the data plane (Stream mode) at one time

Security logs such as traffic and IDP logs are able to be streamed through the traffic interface ports to a remote syslog server. SRX devices do not send streamed session logs to the Routing Engine (RE). Because system logging is performed on the RE, session or traffic logs cannot be written to the RE file system. Therefore, all traffic logging must be sent to a remote syslog server. Because fxp0 belongs to the RE, the remote syslog server must be reachable by an interface on an IOC. Traffic logging cannot be sent out through fxp0.

When the logging mode is set to stream, security traffic logs generated in the data plane are streamed out a revenue traffic port directly to a remote server. That also means your local log file will stop logging. Match condition configuration in System -> Syslog part does not work in Stream mode.  Its as per design, the Routing engine is the one which puts the match condition and filters the log,
since when we use stream mode the traffic is streamed out of the data plane itself and doesn't reach the RE the match condition dose not work when using stream mode and only works in event mode.

Basically, only thing works at System - Syslog section are those generated from control plane.

admin@fw-twn1-1> show configuration security log {     cache;     mode stream;     format sd-syslog;     source-address 10.2.2.13;     stream TO-10-0-0-4 {         format sd-syslog;         category all;         host {             10.0.00.4;         }     }     stream TO-10.4.20.33 {         format sd-syslog;         category all;         host {             10.4.20.33;         }     }     inactive: traceoptions {         file jtac;         flag all;     } }

Control plane pushing configuration to data plane

admin@fw-srx1> show security log Security logging is disabled

“show security log” will show you something about audit log but not policy logging after enabled cache in the security log section, else SRX will show you Security Log disabled.


After you enabled cache under security -> log configuration, as shown at the configuration of section 2.2.2, you will get output like below once you use command show security log:

admin@fw-1> show security log Event time               Message 2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User 'root', command 'xml-mode netconf need-trailer ' 2015-10-02 09:15:04 UTC  UI_LOGOUT_EVENT: User 'root' logout 2015-10-02 09:15:04 UTC  UI_LOGIN_EVENT: User 'root' login, class 'super-user' [55330], ssh-connection '10.4.20.21 7804 10.2.1.14 59097', client-mode 'cli' 2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User 'root', command 'xml-mode netconf need-trailer '

Reference:

• Security logging is disabled
• System Services
• [SRX] Match condition for logging in system syslog does not work when mode in stream and works with event mode
• Stream logging problems in SRX

TekRADIUS is a RADIUS server for Windows with built-in DHCP server. TekRADIUS is tested on Microsoft Windows XP, Vista, Windows 7/8/10 and Windows 2003/2008/2012 server. TekRADIUS complies with RFC 2865 and RFC 2866. TekRADIUS also supports TCP (RFC 6613) and TLS (RFC 6614-RadSec) transports. TekRADIUS has two editions; TekRADIUS(First edition; supports Microsoft SQL Server) and TekRADIUS LT (Second edition; supports SQLite). It runs as a Windows Service and comes with a Win32 management interface.  More feature can be checked from their website.

There are some previous usage posts in my blog:

• TekRADIUS Usage for AD Authentication
• Cisco Switch Radius Configuration Work With TekRADIUS AD Authentication
• Troubleshooting Cisco Radius Configuration with Free Radius Server TekRadius

1. Download and Install TekRADIUS

2.TekRADIUS configuration.

Defined Groups

Radius Client Configuration

Radius Server Configuration

Cisco Attribute

3. JunOS Space Configuration

3.1 Authentication Server Configuration

Authentication Server

3.2 Define a Remote Profile 'admin'

Remote Profile - admin

3.3 Configure JunOS Attribute in TekRadius

3.3.1 Create a new VSA under Vendor Juniper 's attributes list

new vsa - Juniper-Junosspace-Profiles

3.3.2 Assign this new VSA attribute into the group admin

New Success-reply Attribute

4. Verify

4.1 from TekRADIUS server

Here is log from TekRadius server.

4.2 from JunOS Space Audit Logging

Audit Log