Random excessive writes to domain.cfg even though I have VMs turned off?


Callan

Recommended Posts

Open the WebTerminal and execute the following:

while true; do lsof /boot/config/domain.cfg; done;

Wait until you see which process is writing to domain.cfg. This is nothing which should run forever as it produces permanent load. There is no guarantee that it will work, so check the file modification date, too. If it changes and this command didn't return anything, this simply does not work.

Edited by mgutt
Link to comment
  • 8 months later...

So I do use a VM, so turning this off isnt really an option I want to try.  I did try the LSOF loop as stated above and didnt see anything, but the write time of the domain.cfg file most certainly keeps changing .   I am running 6.9.2

Link to comment

You could try the following:

 

Open a web terminal and execute the following:

dd if=/dev/sda of=/dev/null bs=128K iflag=count_bytes count=10G

 

Open a second web terminal and execute the same again:

dd if=/dev/sda of=/dev/null bs=128K iflag=count_bytes count=10G

 

By that we are stressing the usb flash drive so it becomes slower if something tries to write to it.

 

Open a third web terminal and execute our monitoring command:

while true; do lsof /boot/config/domain.cfg; done;

 

Now wait. The idea behind it is, that updating the domain.cfg takes a little bit more time so lsof will be able to catch it.

 

With this method I was able to catch file updates through nano which was not possible without the stressing dd processes:

image.thumb.png.df244dba5fcfeaac030060f881fe83bc.png

Link to comment

ok, but I ak not sure what that has solved... I was able to edit the file while doing the two DD commands and running the lsof in the loop.  Vim barked duriong save because the file was updated during my edit period.  I saved anyways and saw the one item show up in the while loop....   I never saw anything else with an open handle to that file.... 

Link to comment
Posted (edited)

@mgutt

 

ok, so I installed inotify container and see a temp file being written and the copied over the domain.cfg

 


[2021-07-08 20:28:57] test: CONFIGURATION:
[2021-07-08 20:28:57] test: WATCH_DIR=/dir1
[2021-07-08 20:28:57] test: SETTLE_DURATION=5
[2021-07-08 20:28:57] test: MAX_WAIT_TIME=300
[2021-07-08 20:28:57] test: MIN_PERIOD=600
[2021-07-08 20:28:57] test: COMMAND=
[2021-07-08 20:28:57] test: USER_ID=0
[2021-07-08 20:28:57] test: GROUP_ID=0
[2021-07-08 20:28:57] test: UMASK=0600
[2021-07-08 20:28:57] test: DEBUG=True
[2021-07-08 20:28:57] test: USE_POLLING=False
[2021-07-08 20:28:57] test: IGNORE_EVENTS_WHILE_COMMAND_IS_RUNNING=True
[2021-07-08 20:28:57] test: Starting monitor for test
[2021-07-08 20:28:57] test: Using native change detection to detect changes
[2021-07-08 20:28:58] test: Waiting for new change
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_CREATE, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_CREATE, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_MODIFY, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_CREATE, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_CREATE, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_MODIFY, cookie=0, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR3Awnl', wd=4, mask=IN_MOVED_FROM, cookie=400522, name=b'sedR3Awnl'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400522, name=b'domain.cfg'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed4t3UOe', wd=4, mask=IN_MOVED_FROM, cookie=400523, name=b'sed4t3UOe'>
[2021-07-08 20:28:59] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400523, name=b'domain.cfg'>
[2021-07-08 20:28:59] test: Detected change to file /dir1/config/sedR3Awnl
[2021-07-08 20:28:59] test: Waiting for watch directory to stabilize for 5 seconds before triggering command
[2021-07-08 20:29:00] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedo5ctBu', wd=4, mask=IN_CREATE, cookie=0, name=b'sedo5ctBu'>
[2021-07-08 20:29:00] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedo5ctBu', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedo5ctBu'>
[2021-07-08 20:29:00] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedo5ctBu', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedo5ctBu'>
[2021-07-08 20:29:00] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedo5ctBu', wd=4, mask=IN_MOVED_FROM, cookie=400526, name=b'sedo5ctBu'>
[2021-07-08 20:29:00] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400526, name=b'domain.cfg'>
[2021-07-08 20:29:05] test: Watch directory stabilized for 5 seconds. Triggering command.
[2021-07-08 20:29:05] test: Running command with user ID 0, group ID 0, and umask 0600
[2021-07-08 20:29:05] test: vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[2021-07-08 20:29:05] Found existing user "root" with the proper user ID and group ID. Skipping creation of user and group...
[2021-07-08 20:29:05] Running command as user "root"...
Usage: /sbin/setuser USERNAME COMMAND [args..]
[2021-07-08 20:29:05] test: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2021-07-08 20:29:05] test: Finished running command. Exit code was 1
[2021-07-08 20:29:05] test: Waiting for new change
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedmyN7sP', wd=4, mask=IN_CREATE, cookie=0, name=b'sedmyN7sP'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedmyN7sP', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedmyN7sP'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedmyN7sP', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedmyN7sP'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed130Gz2', wd=4, mask=IN_CREATE, cookie=0, name=b'sed130Gz2'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed130Gz2', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sed130Gz2'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed130Gz2', wd=4, mask=IN_MODIFY, cookie=0, name=b'sed130Gz2'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedmyN7sP', wd=4, mask=IN_MOVED_FROM, cookie=400575, name=b'sedmyN7sP'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400575, name=b'domain.cfg'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/sed130Gz2', wd=4, mask=IN_MOVED_FROM, cookie=400577, name=b'sed130Gz2'>
[2021-07-08 20:29:19] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400577, name=b'domain.cfg'>
[2021-07-08 20:29:19] test: Detected change to file /dir1/config/sedmyN7sP
[2021-07-08 20:29:19] test: Waiting for watch directory to stabilize for 5 seconds before triggering command
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_CREATE, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_MOVED_FROM, cookie=400580, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400580, name=b'domain.cfg'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_CREATE, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_ATTRIB, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_MODIFY, cookie=0, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/sedR27qE6', wd=4, mask=IN_MOVED_FROM, cookie=400580, name=b'sedR27qE6'>
[2021-07-08 20:29:20] test: in-event <InotifyEvent: src_path=b'/dir1/config/domain.cfg', wd=4, mask=IN_MOVED_TO, cookie=400580, name=b'domain.cfg'>
[2021-07-08 20:29:25] test: Watch directory stabilized for 5 seconds. Triggering command.
[2021-07-08 20:29:25] test: Command triggered, but it's too soon to run the command again. Waiting another 580 seconds

 

Edited by johntdyer
add @
Link to comment
22 hours ago, johntdyer said:

inotify container

Yes, you can see the file change, but inotify sadly does not return the process.

 

The only option I think is to enable Audit in the Linux Kernel of Unraid and install Audit:

https://slackware.pkgs.org/14.2/slackonly-x86_64/audit-2.3.6-x86_64-1_slonly.txz.html

 

It should return the process:

https://www.cyberciti.biz/tips/linux-audit-files-to-see-who-made-changes-to-a-file.html

 

Link to comment

@mgutt - 

 

Seems to install but I cant run it 

 

 

root@vault:/usr/local/src/audit/audit# auditd -f
Config file /etc/audit/auditd.conf opened for parsing
log_file_parser called with: /var/log/audit/audit.log
log_format_parser called with: RAW
log_group_parser called with: root
priority_boost_parser called with: 4
flush_parser called with: INCREMENTAL
freq_parser called with: 20
num_logs_parser called with: 5
qos_parser called with: lossy
dispatch_parser called with: /sbin/audispd
name_format_parser called with: NONE
max_log_size_parser called with: 6
max_log_size_action_parser called with: ROTATE
space_left_parser called with: 75
space_action_parser called with: SYSLOG
action_mail_acct_parser called with: root
admin_space_left_parser called with: 50
admin_space_left_action_parser called with: SUSPEND
disk_full_action_parser called with: SUSPEND
disk_error_action_parser called with: SUSPEND
tcp_listen_queue_parser called with: 5
tcp_max_per_addr_parser called with: 1
tcp_client_max_idle_parser called with: 0
enable_krb5_parser called with: no
GSSAPI support is not enabled, ignoring value at line 30
krb5_principal_parser called with: auditd
GSSAPI support is not enabled, ignoring value at line 31
Error - audit support not in kernel
Cannot open netlink audit socket
The audit daemon is exiting.
root@vault:/usr/local/src/audit/audit#

 

Link to comment
21 minutes ago, johntdyer said:

Error - audit support not in kernel

As I said. You need to start the Kernel with Audit support. You need to add audit=1 to the Kernel boot options and restart the server (and finally reinstall audit).

 

 

Link to comment
  • 4 weeks later...

@mgutt

 

Tried this and it didnt work.    I added the audit=1 to the append optionsk, restarted the machine, installed the packaghe and it wont start or allow me to watch a file.   Have you actually gotten this to work yourself ? 

 

root@vault:/var/log# auditd  -s enable
Aug  9 07:05:42 vault auditd[8117]: Error - audit support not in kernel
Aug  9 07:05:42 vault auditd[8117]: Cannot open netlink audit socket
Aug  9 07:05:42 vault auditd[8117]: The audit daemon is exiting.
Aug  9 07:05:42 vault auditd: Cannot daemonize (Success)
Aug  9 07:05:42 vault auditd: The audit daemon is exiting.

 

root@vault:/var/log# auditctl  -w /boot/config/domain.cfg
Error - audit support not in kernel
Cannot open netlink audit socket

 

syslinux.cfg

root@vault:/var/log# cat /boot/syslinux/syslinux.cfg
default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append initrd=/bzroot audit=1
label Unraid OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest
root@vault:/var/log#

 

root@vault:/var/log# cat /proc/cmdline
BOOT_IMAGE=/bzimage initrd=/bzroot audit=1
root@vault:/var/log# uptime
 07:08:29 up 42 min,  1 user,  load average: 3.61, 3.56, 3.20
root@vault:/var/log#

 

Link to comment
  • 2 weeks later...
On 7/9/2021 at 9:26 PM, johntdyer said:

Seems to install but I cant run it 

Something which I found by accident:

https://unix.stackexchange.com/a/473972/101920

 

I tried to install sysdig, but it seems it needs to be build for Slackware 14.2 as this package was build for 14.1 and returns errors on installation (even the newer package I found for Slackware Current):

https://slackware.pkgs.org/14.1/slackonly-x86_64/sysdig-0.2.0-x86_64-1_slack.txz.html

wget https://packages.slackonly.com/pub/packages/14.1-x86_64/system/sysdig/sysdig-0.2.0-x86_64-1_slack.txz -P /tmp
upgradepkg --install-new /tmp/sysdig-0.2.0-x86_64-1_slack.txz

 

sysdig needs in addition this package:

wget https://packages.slackonly.com/pub/packages/14.2-x86_64/libraries/jsoncpp/jsoncpp-1.9.1-x86_64-2_slonly.txz -P /tmp
upgradepkg --install-new /tmp/jsoncpp-1.9.1-x86_64-2_slonly.txz

 

 

I tried audit, too, but fails for me as well:

wget -P /tmp https://packages.slackonly.com/pub/packages/14.2-x86_64/system/audit/audit-2.3.6-x86_64-1_slonly.txz
upgradepkg --install-new /tmp/audit-2.3.6-x86_64-1_slonly.txz

cat /boot/syslinux/syslinux.cfg | grep audit
  append initrd=/bzroot audit=1
  
auditctl -f
Error - audit support not in kernel
Cannot open netlink audit socket

auditd -s enable

auditctl  -w /boot/config/domain.cfg
Error - audit support not in kernel
Cannot open netlink audit socket

auditctl -w /boot/config/domain.cfg
Error - audit support not in kernel
Cannot open netlink audit socket

 

But I wonder why you said "dmesg" didn't return anything for you. For me it returns the following:

 dmesg | grep -i audit
[    0.000000] Command line: BOOT_IMAGE=/bzimage initrd=/bzroot audit=1
[    0.098036] Kernel command line: BOOT_IMAGE=/bzimage initrd=/bzroot audit=1
[   34.651587]     audit=1

 

But it seems we can not enable audit as a variable in the Kernel Config was not set by limetech:

https://unix.stackexchange.com/a/507393/101920

Quote

You will need to:

- Download a Kernel that has audit enabled from your distribution provider or compile your Kernel with CONFIG_AUDIT enable if your distribution does not provide such kernel

- If your Kernel is compiled with CONFIG_AUDIT enable(see below) add the kernel parameter audit=1 - See GRUB Quiet Splash. This is the file you need to edit.

 

Then I thought about a third method:

Mount an Unraid path inside of Unraid through FUSE (SSHFS) and limit the bandwidth so lsof would return someting. But sadly Unraid does not include the relevant modules for traffic shaping (tc): 😒

https://forums.unraid.net/topic/78993-solved-how-to-create-a-virtual-nic-for-internalisolated-use-only/?tab=comments#comment-974268

 

 

Link to comment

And another try which failed:

https://sourceware.org/systemtap/SystemTap_Beginners_Guide/inodewatchsect.html

 

I installed the Dev Plugin and selected all packages and installed them. After that I added these packages:

wget -P /tmp https://packages.slackonly.com/pub/packages/14.2-x86_64/development/systemtap/systemtap-4.1-x86_64-1_slonly.txz
upgradepkg --install-new /tmp/systemtap-4.1-x86_64-1_slonly.txz
wget -P /tmp https://slackware.uk/slackware/slackware64-14.2/slackware64/ap/rpm-4.12.0.1-x86_64-1.txz
upgradepkg --install-new /tmp/rpm-4.12.0.1-x86_64-1.txz
wget -P /tmp https://slackware.uk/slackware/slackware64-14.2/patches/packages/mozilla-nss-3.40.1-x86_64-1_slack14.2.txz
upgradepkg --install-new /tmp/mozilla-nss-3.40.1-x86_64-1_slack14.2.txz

 

Then I tried get the inode and execute stap:

stat -c '%D %i' /boot/config/domain.cfg
801 1749
stap inodewatch.stp 0x8 0x01 1749
Checking "/lib/modules/5.10.28-Unraid/build/Module.symvers" failed with error: No such file or directory
Ensure kernel development headers & makefiles are installed

 

I don't understand why it does not work as the Dev Pack includes Headers and Make:

image.png.002d77693c138e6b7dc21dc52ea3ba99.png

 

So tried this tip and created an empty Module.symvers:

touch /lib/modules/5.10.28-Unraid/build/Module.symvers

 

then created:

touch /boot/config/inodewatch.stp

and added the script:

#! /usr/bin/env stap

probe vfs.{write,read}
{
  # dev and ino are defined by vfs.write and vfs.read
  if (dev == MKDEV($1,$2) # major/minor device
      && ino == $3)
    printf ("%s(%d) %s 0x%x/%u\n",
      execname(), pid(), ppfunc(), dev, ino)
}

 

Executed again:

stap /boot/config/inodewatch.stp 0x8 0x01 1749
warning: Generating 12 missing index(es), please wait...
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:1048:19
        source: probe vfs.write = kernel.function("vfs_write")
                                  ^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/5.10.28-Unraid/build'

semantic error: while resolving probe point: identifier 'vfs' at /boot/config/inodewatch.stp:3:7
        source: probe vfs.{write,read}
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]
Number of similar error messages suppressed: 2.
Rerun with -v to see them.

 

I won't give up ^^

Link to comment

@johntdyer

After a little experiment to watch hdparm and smartctl with custom packages, which include a sleep time, I came to the idea, that we could use the same technique to check which process writes to domain.cfg. So I searched for all files which mention "domain.cfg" and accidentally found the following line in /usr/local/emhttp/plugins/dynamix.vm.manager/include/libvirt_helpers.php which updates the domain.cfg on every execution:

        // Read configuration file (guaranteed to exist)
        $domain_cfgfile = "/boot/config/domain.cfg";
        // This will clean any ^M characters (\r) caused by windows from the config file
        shell_exec("sed -i 's!\r!!g' '$domain_cfgfile'");
        $domain_cfg = parse_ini_file($domain_cfgfile);

 

Please use the following command to disable this line:

sed -i '/sed -i/s/^/#/g' /usr/local/emhttp/plugins/dynamix.vm.manager/include/libvirt_helpers.php

 

Wait a little bit and check your USB activity. Didn't solve the problem? Then enable it again:

sed -i '/sed -i/s/^#//g' /usr/local/emhttp/plugins/dynamix.vm.manager/include/libvirt_helpers.php

 

A different file which writes to domain.cfg is /usr/local/emhttp/plugins/dynamix.vm.manager/scripts/libvirtconfig.php, but it should only write to domain.cfg if the VM settings have been changed. But let's verify it by disabling the relevant lines:

if ($cfg_new) {
    foreach ($cfg_new as $key => $value) $tmp .= "$key=\"$value\"\n";
    file_put_contents($cfgfile, $tmp);
}

 

By this command:

sed -i '/file_put_contents/s/^/#/g' /usr/local/emhttp/plugins/dynamix.vm.manager/scripts/libvirtconfig.php

 

Didn't help? Enable it again:

sed -i '/file_put_contents/s/^#//g' /usr/local/emhttp/plugins/dynamix.vm.manager/scripts/libvirtconfig.php

 

I did not found other files, so I cross my fingers that one of these commands solves your problem.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.