If you use XCP (or have used it), you might have run into an issue or two on occasion – generally related to memory issues or problems with XCP sync taking too long.
The new XCP 1.6.1 release aims to fix some of those issues.
Key enhancements
Sync Performance : Identified sync bottlenecks w.r.t change handling and fixed in 1.6.1 to Improve the sync performance. Now incremental sync takes less time than baseline. This would help performing more frequent syncs or will need less sync/cutover window.
Memory Optimization : Memory allocation was a challenge in the previous versions and used to receive “Memory allocation” errors during the large sync jobs. XCP 1.6.1 handles the memory allocation elegantly to further assist in accomplishing the sync job.
Syslog integration : Configure Syslog option to send XCP events to the remote syslog server for monitoring the XCP jobs.
Complete Features List of NetApp XCP 1.6.1
- Sync performance improvements (NFS/SMB)
- Memory optimization (NFS Sync)
- File Analytics (General Availability)
- Event logging – XCP now supports logging events to xcp_event.log file
- Syslog support (NFS/SMB)
The XCP TME Karthik Nagalingam also wrote up a blog on XCP:
https://blog.netapp.com/data-migration-xcp
File Analytics
In XCP 1.6, a new File Analytics Engine was added to XCP. This basically configures an Apache web server, a postgresql database and adds a web GUI to view file information using an XCP listener agent. Karthik also wrote a blog on that:
https://blog.netapp.com/xcp-data-migration-software
With this feature, you can add your NFS server/volume and XCP will run periodic scans on the datasets to provide information.
Once you add a server, you get a list of the available exports:
Once you have the shares, you can click on one and run a scan:
Once that’s done, you can access the Stat View and File Distribution tabs:
The stat view includes:
- Hot/cold data size and count
- File capacity sorted by size
- File distribution by size
- Directory depth
- Directory entries
- Accessed time
- Information on space occupied by users
Postgresql DB
In addition to the web GUI, you also can access the postgresql database. That way you can query the database if you choose.
When XCP configures the database, a new table called xcp_cmdb is added:
# sudo -u postgres psql
psql (9.2.24)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
xcp_cmdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)
You can access that database and query which tables are added:
# sudo -u postgres psql xcp_cmdb
psql (9.2.24)
Type "help" for help.
xcp_cmdb=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------+-------+---------
public | xcp_agent | table | central
public | xcp_login | table | central
public | xcp_nfs_file_server | table | central
public | xcp_nfs_jobs | table | central
public | xcp_nfs_stats | table | central
public | xcp_nfs_tree | table | central
public | xcp_smb_file_server | table | central
public | xcp_smb_jobs | table | central
public | xcp_smb_stats | table | central
public | xcp_smb_tree | table | central
(10 rows)
You can use standard Postgresql commands to query the tables.
xcp_cmdb=# SELECT file_server_ip FROM xcp_nfs_file_server;
file_server_ip
----------------
10.10.10.10
(1 row)
xcp_cmdb=# SELECT * FROM xcp_nfs_jobs;
id | export_path | agent_id | process_group_id | start_timestamp |
end_timestamp | type | status | statistics |
stdout | stderr | log | protocol | files_to_process
--------------------------------------+-------------------------+--------------------------------------+------------------+----------------------------+-----
-----------------------+------+----------+------------------------------------------------------------------------------------------------------------------+
--------+--------+-----+----------+------------------
745a0615-4177-4d83-b945-8e302efc89e9 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 6562 | 2020-07-27 06:26:55.980298 | 2020
-07-27 06:26:56.785463 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 138247, "sent": 106400, "receive_throughput": 2208273} |
stdout | stderr | log | NFS | 9047
9a5fc78e-7260-47dd-8cb5-da1820dbcd3e | 10.10.10.10:/home | bf83aced-81cd-4b7f-be5d-94e55df11484 | 2039 | 2020-07-31 00:34:50.001866 | 2020
-07-31 00:34:50.390331 | SCAN | FINISHED | {"received": 9808, "scanned": 21, "sent_throughput": 5627, "sent": 2072, "receive_throughput": 26638} |
stdout | stderr | log | NFS | 839
2767d48c-be49-4232-9a49-9632878bd4c6 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 5832 | 2020-07-27 06:17:47.496321 | 2020
-07-27 06:17:48.290484 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 83533, "sent": 106400, "receive_throughput": 1334304} |
stdout | stderr | log | NFS | 9114
(3 rows)
Installation/Upgrade Note
If you’re already running an XCP version prior to 1.6.1, you’ll need to re-run the configure.sh script. This creates the XCP service module on the client (NFS version) to provide better overall management.
For example, if I simply untar the XCP install (like you would do in prior versions to upgrade), then XCP commands work fine, but the new File Systems Analytics module doesn’t, as it needs the XCP service to run properly.
# tar -xvf NETAPP_XCP_1.6.1.tgz
./
./xcp/
./xcp/linux/
./xcp/linux/xcp
./xcp/windows/
./xcp/windows/xcp.exe
./xcp/xcp_gui/
./xcp/xcp_gui/xcp/
./xcp/configure.sh
./xcp/README
# xcp activate
XCP 1.6.1; (c) 2020 NetApp, Inc.; Licensed to Justin Parisi [NetApp Inc] until Thu Oct 22 12:31:11 2020
XCP already activated
Trying to start the listener will fail:
# ./linux/xcp --listen &
[1] 1525
[root@XCP xcp]# Traceback (most recent call last):
File "../xcp.py", line 957, in <module>
File "../xcp.py", line 902, in main
File "../rest/config.py", line 117, in <module>
File "../rest/config.py", line 112, in singleton
File "../rest/config.py", line 163, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3215, in first
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 354, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
File "/local/build/lib/python2.7/site-packages/pgdb.py", line 1623, in connect
sqlalchemy.exc.InternalError: (pg.InternalError) could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
(Background on this error at: http://sqlalche.me/e/2j85)
[1]+ Exit 1 ./linux/xcp --listen
When we look for the XCP service, it’s not there. That gets created with the script:
# service xcp status
Redirecting to /bin/systemctl status xcp.service
Unit xcp.service could not be found.
# systemctl status xcp
Unit xcp.service could not be found.
# sudo systemctl start xcp
Failed to start xcp.service: Unit not found.
Here’s an example of the configure.sh script to upgrade:
# ./configure.sh
------------XCP CONFIGURATION SCRIPT -------------
# Checking if xcp service is running
Menu Help:
# Install/ Upgrade -
# Install - Installs PostgreSQL and httpd if not installed and does setup for XCP File Analytics.
# Upgrade - Upgrades takes a backup of xcp folder under /var/www/html/ to xcp_backup
# Repair - Restarts PostgreSQL database, http and XCP services incase if any of them are stopped.
# Cleanup - Uninstalls PostgreSQL and httpd.
1. Installation/Upgrade
2. Repair
3. Cleanup
0. Exit
Please select the option from the menu [0-3]: 1
--------Setting up Postgres -----------
# Checking if postgres is installed
Redirecting to /bin/systemctl status postgresql.service
# Postgres service is already running
--------------Setting up httpd -------------
# Checking if httpd is installed
httpd-2.4.6-93.el7.centos.x86_64
# HTTPD is already installed...
--------------Setting up https -------------
mod_ssl-2.4.6-93.el7.centos.x86_64
# Https is already installed...
--------------Configuring xcp --------------
# Copying xcp files.....
# Taking backup of xcp folder /var/www/html/xcp in /var/www/html/xcp_backup
NOTE:
If you are running for first time, select option 1.
If you are upgrading select option 0.
______________________________
Please choose the menu you want to start:
1. Configure client system
2. Reset admin password
3. Reset Postgres database password
4. Rebuild xcp.ini file
5. Delete database and reconfigure client system
0. Quit
-------------------------------
Enter digit between [0-5] >> 1
# Postgres service is running
# Database already exists and configured.
---------------Starting xcp service -------------
# Creating xcp service...
Redirecting to /bin/systemctl status xcp.service
# XCP service started successfully
Created symlink from /etc/systemd/system/multi-user.target.wants/xcp.service to /etc/systemd/system/xcp.service.
root 1818 1 19 20:03 ? 00:00:00 /usr/bin/xcp --listen
postgres 1833 1587 0 20:03 ? 00:00:00 postgres: central xcp_cmdb 127.0.0.1(34018) idle
---------------------------------------------
XCP File analytics is configured successfully
To start XCP service manually, use "sudo systemctl start xcp" command
Use following link and login as 'admin' user for GUI access
https://XCP/xcp
---------------------------------------------
Once that’s done, XCP is ready to use!
Full NetApp Support
Since XCP 1.5, you can call in for official NetApp Support for XCP.
Support Site Self-Service Options
To review, order, and renew software subscriptions, click My Service Contracts & Warranties on the Service and Support page of the NetApp Support site.
